Request 6522 (accepted)

Overview

Request 6522 (accepted)

Update to version 4.2

Created by Aloysius 4 days ago
In state accepted

Submit x265

Submit package home:Aloysius:branches:Staging / x265 to package Staging / x265

x265.changes Changed

@@ -1,4 +1,61 @@
 -------------------------------------------------------------------
+Sat May  9 11:26:50 UTC 2026 - Luigi Baldoni <aloisio@gmx.com>
+
+- Update to version 4.2
+  New feature:
+  * Threaded Motion Estimation (Experimental feature)-It uses a
+    dedicated threadpool to precompute Motion Estimation in
+    parallel.Improves encoding speed upto 1.5x for 1080p & lower
+    resolution on multi core machines with low frequency setting.
+    On high frequency systems or on machines with low number of
+    cores, the overhead of additional Motion estimation work may
+    outweigh parallelism
+  Enhancements to existing features:
+  * Add new Levels 6.3 to 7.2 specified in ITU-T H.265 (V9)
+    (09/2023) and above
+  * Improve Slices feature with check zeroMv
+  * Enable frame parallelism with MCSTF feature
+  * Updated support to signal AOM FGM params
+  * Improve quality with SBRC feature
+  * Updated DolbyVision P5 VUI defaults
+  API changes:
+  * API Support to enable Threaded Motion
+    Estimation(--threaded-me)
+  Optimizations:
+  * RISC V optimizations including SAD, SATD, DCT, IDCT, block
+    copy, pixel utilities, SAO, loopfilter, transpose kernels
+    resulting in 2x encoding speed.
+  * ARM SIMD optimizations including the use of NEON and SVE
+    instruction set extensions. The following algorithms now
+    have optimized SIMD implementations: DST, IDCT, SSE, SSD,
+    intra_pred_planar, pelFilterLumaStrong, interpolation,
+    planecopy, dequant_normal, blockcopy, pixel variance
+    resulting in 8% faster encoding speed compared to v4.1
+  Bug fixes:
+  * Fix memory leaks (no command line option, SEI buffer,
+    analysis save/load)
+  * Fix chroma qp offset for non yuv444 inputs
+  * Fix max supported input resolution
+  * Fix bugs with ARM SIMD optimizations
+  * Fix Alpha and Multiview feature flag support in x265_config
+  * Fix test harness issues, CMake errors
+  * Fix inconsistent output with aq-motion
+  * Fix crash with hist-scenecut on high bit-depth builds
+  * Fix lookahead concurrency bug
+  * Fix shared link issue (R_X86_64_PC32), yuv recon output
+     issue, rd-refine and dynamic-refine issue, inputs for
+     Windows named pipe,weighted prediction delta_chroma_offset,
+     crf and vbv issue in abr-ladder, psnr and ssim reported
+     with MCSTF feature, internally overflowed VBV variables
+  Known issues:
+  * Output mismatch between analysis save & load with cutree
+    with reuse level < 10
+  * Inconsistent hash mismatch with abr-ladder feature
+  * Performance regression observed with threaded-me feature on
+    high frequency systems and for higher resolutions (4k)
+- Reworked x265.patch
+
+-------------------------------------------------------------------
 Mon Mar  3 03:03:03 UTC 2025 - olaf@aepfle.de
 
 - Update to version 4.1

x265.spec Changed

x265.patch Changed

_service Changed

baselibs.conf Changed

x265-4.1.tar/source/common/aarch64/blockcopy8-common.S Deleted

@@ -1,54 +0,0 @@
-/*****************************************************************************
- * Copyright (C) 2022-2023 MulticoreWare, Inc
- *
- * Authors: David Chen <david.chen@myais.com.cn>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
- *
- * This program is also available under a commercial proprietary license.
- * For more information, contact us at license @ x265.com.
- *****************************************************************************/
-
-// This file contains the macros written using NEON instruction set
-// that are also used by the SVE2 functions
-
-#include "asm.S"
-
-.arch           armv8-a
-
-// void cpy1Dto2D_shr(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
-.macro cpy1Dto2D_shr_start
-    add             x2, x2, x2
-    dup             v0.8h, w3
-    cmeq            v1.8h, v1.8h, v1.8h
-    sshl            v1.8h, v1.8h, v0.8h
-    sri             v1.8h, v1.8h, #1
-    neg             v0.8h, v0.8h
-.endm
-
-.macro cpy2Dto1D_shr_start
-    add             x2, x2, x2
-    dup             v0.8h, w3
-    cmeq            v1.8h, v1.8h, v1.8h
-    sshl            v1.8h, v1.8h, v0.8h
-    sri             v1.8h, v1.8h, #1
-    neg             v0.8h, v0.8h
-.endm
-
-const xtn_xtn2_table, align=4
-.byte    0, 2, 4, 6, 8, 10, 12, 14
-.byte    16, 18, 20, 22, 24, 26, 28, 30
-endconst
-

x265-4.1.tar/source/common/aarch64/blockcopy8-sve.S Deleted

@@ -1,1416 +0,0 @@
-/*****************************************************************************
- * Copyright (C) 2022-2023 MulticoreWare, Inc
- *
- * Authors: David Chen <david.chen@myais.com.cn>
- 
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
- *
- * This program is also available under a commercial proprietary license.
- * For more information, contact us at license @ x265.com.
- *****************************************************************************/
-
-#include "asm-sve.S"
-#include "blockcopy8-common.S"
-
-.arch armv8-a+sve
-
-#ifdef __APPLE__
-.section __RODATA,__rodata
-#else
-.section .rodata
-#endif
-
-.align 4
-
-.text
-
-/* void blockcopy_sp(pixel* a, intptr_t stridea, const int16_t* b, intptr_t strideb)
- *
- * r0   - a
- * r1   - stridea
- * r2   - b
- * r3   - strideb */
-
-function PFX(blockcopy_sp_4x4_sve)
-    ptrue           p0.h, vl4
-.rept 2
-    ld1h            {z0.h}, p0/z, x2
-    add             x2, x2, x3, lsl #1
-    st1b            {z0.h}, p0, x0
-    add             x0, x0, x1
-    ld1h            {z1.h}, p0/z, x2
-    add             x2, x2, x3, lsl #1
-    st1b            {z1.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_8x8_sve)
-    ptrue           p0.h, vl8
-.rept 4
-    ld1h            {z0.h}, p0/z, x2
-    add             x2, x2, x3, lsl #1
-    st1b            {z0.h}, p0, x0
-    add            x0, x0, x1
-    ld1h            {z1.h}, p0/z, x2
-    add             x2, x2, x3, lsl #1
-    st1b            {z1.h}, p0, x0
-    add            x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_16x16_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_sp_16_16
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.rept 8
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    st1             {v0.16b}, x0, x1
-    st1             {v1.16b}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_sp_16_16:
-    ptrue           p0.h, vl16
-.rept 8
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-    ld1h            {z1.h}, p0/z, x2
-    st1b            {z1.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_32x32_sve)
-    mov             w12, #4
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_sp_32_32
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.Loop_csp32_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.8h-v3.8h}, x2, x3
-    ld1             {v4.8h-v7.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    tbl             v2.16b, {v4.16b,v5.16b}, v31.16b
-    tbl             v3.16b, {v6.16b,v7.16b}, v31.16b
-    st1             {v0.16b-v1.16b}, x0, x1
-    st1             {v2.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_csp32_sve
-    ret
-.vl_gt_16_blockcopy_sp_32_32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_sp_32_32
-    ptrue           p0.h, vl16
-.vl_gt_16_loop_csp32_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1h            {z0.h}, p0/z, x2
-    ld1h            {z1.h}, p0/z, x2, #1, mul vl
-    st1b            {z0.h}, p0, x0
-    st1b            {z1.h}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-    ld1h            {z2.h}, p0/z, x2
-    ld1h            {z3.h}, p0/z, x2, #1, mul vl
-    st1b            {z2.h}, p0, x0
-    st1b            {z3.h}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    cbnz            w12, .vl_gt_16_loop_csp32_sve
-    ret
-.vl_gt_48_blockcopy_sp_32_32:
-    ptrue           p0.h, vl32
-.vl_gt_48_loop_csp32_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-    ld1h            {z1.h}, p0/z, x2
-    st1b            {z1.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    cbnz            w12, .vl_gt_48_loop_csp32_sve
-    ret
-endfunc
-
-function PFX(blockcopy_ps_16x16_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ps_16_16
-    lsl             x1, x1, #1
-.rept 8
-    ld1             {v4.16b}, x2, x3
-    ld1             {v5.16b}, x2, x3
-    uxtl            v0.8h, v4.8b
-    uxtl2           v1.8h, v4.16b
-    uxtl            v2.8h, v5.8b
-    uxtl2           v3.8h, v5.16b
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_ps_16_16:
-    ptrue           p0.b, vl32
-.rept 16
-    ld1b            {z1.h}, p0/z, x2
-    st1h            {z1.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_32x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ps_32_32
-    lsl             x1, x1, #1
-    mov             w12, #4
-.Loop_cps32_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v17.16b}, x2, x3
-    ld1             {v18.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, x1
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps32_sve
-    ret
-.vl_gt_16_blockcopy_ps_32_32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ps_32_32
-    ptrue           p0.b, vl32
-.rept 32
-    ld1b            {z2.h}, p0/z, x2
-    ld1b            {z3.h}, p0/z, x2, #1, mul vl
-    st1h            {z2.h}, p0, x0
-    st1h            {z3.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-.vl_gt_48_blockcopy_ps_32_32:
-    ptrue           p0.b, vl64
-.rept 32
-    ld1b            {z2.h}, p0/z, x2
-    st1h            {z2.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_64x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ps_64_64
-    lsl             x1, x1, #1
-    sub             x1, x1, #64
-    mov             w12, #16
-.Loop_cps64_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, #64
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps64_sve
-    ret
-.vl_gt_16_blockcopy_ps_64_64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ps_64_64
-    ptrue           p0.b, vl32
-.rept 64
-    ld1b            {z4.h}, p0/z, x2
-    ld1b            {z5.h}, p0/z, x2, #1, mul vl
-    ld1b            {z6.h}, p0/z, x2, #2, mul vl
-    ld1b            {z7.h}, p0/z, x2, #3, mul vl
-    st1h            {z4.h}, p0, x0
-    st1h            {z5.h}, p0, x0, #1, mul vl
-    st1h            {z6.h}, p0, x0, #2, mul vl
-    st1h            {z7.h}, p0, x0, #3, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-.vl_gt_48_blockcopy_ps_64_64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_blockcopy_ps_64_64
-    ptrue           p0.b, vl64
-.rept 64
-    ld1b            {z4.h}, p0/z, x2
-    ld1b            {z5.h}, p0/z, x2, #1, mul vl
-    st1h            {z4.h}, p0, x0
-    st1h            {z5.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-.vl_gt_112_blockcopy_ps_64_64:
-    ptrue           p0.b, vl128
-.rept 64
-    ld1b            {z4.h}, p0/z, x2
-    st1h            {z4.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-
-endfunc
-
-function PFX(blockcopy_ss_16x16_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ss_16_16
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 8
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_ss_16_16:
-    ptrue           p0.h, vl16
-.rept 16
-    ld1h            {z0.h}, p0/z, x2
-    st1h            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_32x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ss_32_32
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-    mov             w12, #4
-.Loop_css32_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css32_sve
-    ret
-.vl_gt_16_blockcopy_ss_32_32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ss_32_32
-    ptrue           p0.h, vl16
-.rept 32
-    ld1h            {z0.h}, p0/z, x2
-    ld1h            {z1.h}, p0/z, x2, #1, mul vl
-    st1h            {z0.h}, p0, x0
-    st1h            {z1.h}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-.vl_gt_48_blockcopy_ss_32_32:
-    ptrue           p0.h, vl32
-.rept 32
-    ld1h            {z0.h}, p0/z, x2
-    st1h            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_64x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ss_64_64
-    lsl             x1, x1, #1
-    sub             x1, x1, #64
-    lsl             x3, x3, #1
-    sub             x3, x3, #64
-    mov             w12, #8
-.Loop_css64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, #64
-    ld1             {v4.8h-v7.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, #64
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css64_sve
-    ret
-.vl_gt_16_blockcopy_ss_64_64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ss_64_64
-    mov             w12, #8
-    ptrue           p0.b, vl32
-.vl_gt_16_loop_css64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    ld1b            {z1.b}, p0/z, x2, #1, mul vl
-    ld1b            {z2.b}, p0/z, x2, #2, mul vl
-    ld1b            {z3.b}, p0/z, x2, #3, mul vl
-    st1b            {z0.b}, p0, x0
-    st1b            {z1.b}, p0, x0, #1, mul vl
-    st1b            {z2.b}, p0, x0, #2, mul vl
-    st1b            {z3.b}, p0, x0, #3, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    cbnz            w12, .vl_gt_16_loop_css64_sve
-    ret
-.vl_gt_48_blockcopy_ss_64_64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_blockcopy_ss_64_64
-    mov             w12, #8
-    ptrue           p0.b, vl64
-.vl_gt_48_loop_css64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    ld1b            {z1.b}, p0/z, x2, #1, mul vl
-    st1b            {z0.b}, p0, x0
-    st1b            {z1.b}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    cbnz            w12, .vl_gt_48_loop_css64_sve
-    ret
-.vl_gt_112_blockcopy_ss_64_64:
-    mov             w12, #8
-    ptrue           p0.b, vl128
-.vl_gt_112_loop_css64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    st1b            {z0.b}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    cbnz            w12, .vl_gt_112_loop_css64_sve
-    ret
-endfunc
-
-/******** Chroma blockcopy********/
-function PFX(blockcopy_ss_16x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ss_16_32
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 16
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_ss_16_32:
-    ptrue           p0.h, vl16
-.rept 32
-    ld1h            {z0.h}, p0/z, x2
-    st1h            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_32x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ss_32_64
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-    mov             w12, #8
-.Loop_css32x64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css32x64_sve
-    ret
-.vl_gt_16_blockcopy_ss_32_64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ss_32_64
-    mov             w12, #8
-    ptrue           p0.b, vl32
-.vl_gt_32_loop_css32x64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    ld1b            {z1.b}, p0/z, x2, #1, mul vl
-    st1b            {z0.b}, p0, x0
-    st1b            {z1.b}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    cbnz            w12, .vl_gt_32_loop_css32x64_sve
-    ret
-.vl_gt_48_blockcopy_ss_32_64:
-    mov             w12, #8
-    ptrue           p0.b, vl64
-.vl_gt_48_loop_css32x64_sve:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    st1b            {z0.b}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1, lsl #1
-.endr
-    cbnz            w12, .vl_gt_48_loop_css32x64_sve
-    ret
-endfunc
-
-// chroma blockcopy_ps
-function PFX(blockcopy_ps_4x8_sve)
-    ptrue           p0.h, vl4
-.rept 8
-    ld1b            {z0.h}, p0/z, x2
-    st1h            {z0.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_8x16_sve)
-    ptrue           p0.h, vl8
-.rept 16
-    ld1b            {z0.h}, p0/z, x2
-    st1h            {z0.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_16x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ps_16_32
-    lsl             x1, x1, #1
-.rept 16
-    ld1             {v4.16b}, x2, x3
-    ld1             {v5.16b}, x2, x3
-    uxtl            v0.8h, v4.8b
-    uxtl2           v1.8h, v4.16b
-    uxtl            v2.8h, v5.8b
-    uxtl2           v3.8h, v5.16b
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_ps_16_32:
-    ptrue           p0.b, vl32
-.rept 32
-    ld1b            {z1.h}, p0/z, x2
-    st1h            {z1.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_32x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_ps_32_64
-    lsl             x1, x1, #1
-    mov             w12, #8
-.Loop_cps32x64_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v17.16b}, x2, x3
-    ld1             {v18.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, x1
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps32x64_sve
-    ret
-.vl_gt_16_blockcopy_ps_32_64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_ps_32_64
-    ptrue           p0.b, vl32
-.rept 64
-    ld1b            {z2.h}, p0/z, x2
-    ld1b            {z3.h}, p0/z, x2, #1, mul vl
-    st1h            {z2.h}, p0, x0
-    st1h            {z3.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-.vl_gt_48_blockcopy_ps_32_64:
-    ptrue           p0.b, vl64
-.rept 64
-    ld1b            {z2.h}, p0/z, x2
-    st1h            {z2.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3
-.endr
-    ret
-endfunc
-
-// chroma blockcopy_sp
-function PFX(blockcopy_sp_4x8_sve)
-    ptrue           p0.h, vl4
-.rept 8
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_8x16_sve)
-    ptrue           p0.h, vl8
-.rept 16
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_16x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_sp_16_32
-    ptrue           p0.h, vl8
-.rept 32
-    ld1h            {z0.h}, p0/z, x2
-    ld1h            {z1.h}, p0/z, x2, #1, mul vl
-    st1b            {z0.h}, p0, x0
-    st1b            {z1.h}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_sp_16_32:
-    ptrue           p0.h, vl16
-.rept 32
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_32x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_sp_32_64
-    ptrue           p0.h, vl8
-.rept 64
-    ld1h            {z0.h}, p0/z, x2
-    ld1h            {z1.h}, p0/z, x2, #1, mul vl
-    ld1h            {z2.h}, p0/z, x2, #2, mul vl
-    ld1h            {z3.h}, p0/z, x2, #3, mul vl
-    st1b            {z0.h}, p0, x0
-    st1b            {z1.h}, p0, x0, #1, mul vl
-    st1b            {z2.h}, p0, x0, #2, mul vl
-    st1b            {z3.h}, p0, x0, #3, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_sp_32_64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_sp_32_64
-    ptrue           p0.h, vl16
-.rept 64
-    ld1h            {z0.h}, p0/z, x2
-    ld1h            {z1.h}, p0/z, x2, #1, mul vl
-    st1b            {z0.h}, p0, x0
-    st1b            {z1.h}, p0, x0, #1, mul vl
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_48_blockcopy_sp_32_64:
-    ptrue           p0.h, vl32
-.rept 64
-    ld1h            {z0.h}, p0/z, x2
-    st1b            {z0.h}, p0, x0
-    add             x2, x2, x3, lsl #1
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-/* blockcopy_pp(pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride) */
-
-function PFX(blockcopy_pp_32x8_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_pp_32_8
-.rept 8
-    ld1             {v0.16b-v1.16b}, x2, x3
-    st1             {v0.16b-v1.16b}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockcopy_pp_32_8:
-    ptrue           p0.b, vl32
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    st1b            {z0.b}, p0, x0
-    add             x2, x2, x3
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-.macro blockcopy_pp_32xN_sve h
-function PFX(blockcopy_pp_32x\h\()_sve)
-    mov             w12, #\h / 8
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_pp_32xN_\h
-.Loop_sve_32x\h\():
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.16b-v1.16b}, x2, x3
-    st1             {v0.16b-v1.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_sve_32x\h
-    ret
-.vl_gt_16_blockcopy_pp_32xN_\h:
-    ptrue           p0.b, vl32
-.L_gt_16_blockcopy_pp_32xN_\h:
-    sub             w12, w12, #1
-.rept 8
-    ld1b            {z0.b}, p0/z, x2
-    st1b            {z0.b}, p0, x0
-    add             x2, x2, x3
-    add             x0, x0, x1
-.endr
-    cbnz            w12, .L_gt_16_blockcopy_pp_32xN_\h
-    ret
-endfunc
-.endm
-
-blockcopy_pp_32xN_sve 16
-blockcopy_pp_32xN_sve 24
-blockcopy_pp_32xN_sve 32
-blockcopy_pp_32xN_sve 64
-blockcopy_pp_32xN_sve 48
-
-.macro blockcopy_pp_64xN_sve h
-function PFX(blockcopy_pp_64x\h\()_sve)
-    mov             w12, #\h / 4
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockcopy_pp_64xN_\h
-.Loop_sve_64x\h\():
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.16b-v3.16b}, x2, x3
-    st1             {v0.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_sve_64x\h
-    ret
-.vl_gt_16_blockcopy_pp_64xN_\h:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockcopy_pp_64xN_\h
-    ptrue           p0.b, vl32
-.L_le_32_blockcopy_pp_64xN_\h:
-    sub             w12, w12, #1
-.rept 4
-    ld1b            {z0.b}, p0/z, x2
-    ld1b            {z1.b}, p0/z, x2, #1, mul vl
-    st1b            {z0.b}, p0, x0
-    st1b            {z1.b}, p0, x0, #1, mul vl
-    add             x2, x2, x3
-    add             x0, x0, x1
-.endr
-    cbnz            w12, .L_le_32_blockcopy_pp_64xN_\h
-    ret
-.vl_gt_48_blockcopy_pp_64xN_\h:
-    ptrue           p0.b, vl64
-.L_blockcopy_pp_64xN_\h:
-    sub             w12, w12, #1
-.rept 4
-    ld1b            {z0.b}, p0/z, x2
-    st1b            {z0.b}, p0, x0
-    add             x2, x2, x3
-    add             x0, x0, x1
-.endr
-    cbnz            w12, .L_blockcopy_pp_64xN_\h
-    ret
-endfunc
-.endm
-
-blockcopy_pp_64xN_sve 16
-blockcopy_pp_64xN_sve 32
-blockcopy_pp_64xN_sve 48
-blockcopy_pp_64xN_sve 64
-
-function PFX(blockfill_s_32x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_blockfill_s_32_32
-    dup             v0.8h, w2
-    mov             v1.16b, v0.16b
-    mov             v2.16b, v0.16b
-    mov             v3.16b, v0.16b
-    lsl             x1, x1, #1
-.rept 32
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    ret
-.vl_gt_16_blockfill_s_32_32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_blockfill_s_32_32
-    dup             z0.h, w2
-    ptrue           p0.h, vl16
-.rept 32
-    st1h            {z0.h}, p0, x0
-    st1h            {z0.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-.vl_gt_48_blockfill_s_32_32:
-    dup             z0.h, w2
-    ptrue           p0.h, vl32
-.rept 32
-    st1h            {z0.h}, p0, x0
-    add             x0, x0, x1, lsl #1
-.endr
-    ret
-endfunc
-
-// void cpy2Dto1D_shl(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
-.macro cpy2Dto1D_shl_start_sve
-    add             x2, x2, x2
-    mov             z0.h, w3
-.endm
-
-function PFX(cpy2Dto1D_shl_16x16_sve)
-    dup             z0.h, w3
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy2Dto1D_shl_16x16
-    cpy2Dto1D_shl_start_sve
-    mov             w12, #4
-.Loop_cpy2Dto1D_shl_16_sve:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.16b-v3.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b-v3.16b}, x0, #32
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_16_sve
-    ret
-.vl_gt_16_cpy2Dto1D_shl_16x16:
-    ptrue           p0.h, vl16
-.rept 16
-    ld1h            {z1.h}, p0/z, x1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0
-    add             x1, x1, x2, lsl #1
-    add             x0, x0, #32
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_32x32_sve)
-    dup             z0.h, w3
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy2Dto1D_shl_32x32
-    cpy2Dto1D_shl_start_sve
-    mov             w12, #16
-.Loop_cpy2Dto1D_shl_32_sve:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_32_sve
-    ret
-.vl_gt_16_cpy2Dto1D_shl_32x32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_cpy2Dto1D_shl_32x32
-    ptrue           p0.h, vl16
-.rept 32
-    ld1h            {z1.h}, p0/z, x1
-    ld1h            {z2.h}, p0/z, x1, #1, mul vl
-    lsl             z1.h, p0/m, z1.h, z0.h
-    lsl             z2.h, p0/m, z2.h, z0.h
-    st1h            {z1.h}, p0, x0
-    st1h            {z2.h}, p0, x0, #1, mul vl
-    add             x1, x1, x2, lsl #1
-    add             x0, x0, #64
-.endr
-    ret
-.vl_gt_48_cpy2Dto1D_shl_32x32:
-    ptrue           p0.h, vl32
-.rept 32
-    ld1h            {z1.h}, p0/z, x1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0
-    add             x1, x1, x2, lsl #1
-    add             x0, x0, #64
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_64x64_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy2Dto1D_shl_64x64
-    cpy2Dto1D_shl_start_sve
-    mov             w12, #32
-    sub             x2, x2, #64
-.Loop_cpy2Dto1D_shl_64_sve:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    ld1             {v16.16b-v19.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    sshl            v16.8h, v16.8h, v0.8h
-    sshl            v17.8h, v17.8h, v0.8h
-    sshl            v18.8h, v18.8h, v0.8h
-    sshl            v19.8h, v19.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-    st1             {v16.16b-v19.16b}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_64_sve
-    ret
-.vl_gt_16_cpy2Dto1D_shl_64x64:
-    dup             z0.h, w3
-    mov             x8, #64
-    mov             w12, #64
-.L_init_cpy2Dto1D_shl_64x64:
-    sub             w12, w12, 1
-    mov             x9, #0
-    whilelt         p0.h, x9, x8
-.L_cpy2Dto1D_shl_64x64:
-    ld1h            {z1.h}, p0/z, x1, x9, lsl #1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0, x9, lsl #1
-    inch            x9
-    whilelt         p0.h, x9, x8
-    b.first         .L_cpy2Dto1D_shl_64x64
-    add             x1, x1, x2, lsl #1
-    addvl           x0, x0, #1
-    cbnz            w12, .L_init_cpy2Dto1D_shl_64x64
-    ret
-endfunc
-
-// void cpy2Dto1D_shr(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
-
-function PFX(cpy2Dto1D_shr_4x4_sve)
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    ptrue           p0.h, vl8
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-    lsl             x2, x2, #1
-    index           z3.d, #0, x2
-    index           z4.d, #0, #8
-.rept 2
-    ld1d            {z5.d}, p0/z, x1, z3.d
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0, z4.d
-    add             x0, x0, #16
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_8x8_sve)
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    ptrue           p0.h, vl8
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 8
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, #16
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_16x16_sve)
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy2Dto1D_shr_16x16
-    ptrue           p0.h, vl8
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 16
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    add             x0, x0, #32
-.endr
-    ret
-.vl_gt_16_cpy2Dto1D_shr_16x16:
-    ptrue           p0.h, vl16
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 16
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, #32
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_32x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy2Dto1D_shr_32x32
-    cpy2Dto1D_shr_start
-    mov             w12, #16
-.Loop_cpy2Dto1D_shr_32_sve:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.8h-v5.8h}, x1, x2
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sub             v4.8h, v4.8h, v1.8h
-    sub             v5.8h, v5.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.8h-v5.8h}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shr_32_sve
-    ret
-.vl_gt_16_cpy2Dto1D_shr_32x32:
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    cmp             x9, #48
-    bgt             .vl_gt_48_cpy2Dto1D_shr_32x32
-    ptrue           p0.h, vl16
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 32
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    add             x0, x0, #64
-.endr
-    ret
-.vl_gt_48_cpy2Dto1D_shr_32x32:
-    ptrue           p0.h, vl32
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 32
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, x2, lsl #1
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, #64
-.endr
-    ret
-endfunc
-
-// void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
-
-function PFX(cpy1Dto2D_shl_16x16_sve)
-    dup             z0.h, w3
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy1Dto2D_shl_16x16
-    ptrue           p0.h, vl8
-.rept 16
-    ld1h            {z1.h}, p0/z, x1
-    ld1h            {z2.h}, p0/z, x1, #1, mul vl
-    lsl             z1.h, p0/m, z1.h, z0.h
-    lsl             z2.h, p0/m, z2.h, z0.h
-    st1h            {z1.h}, p0, x0
-    st1h            {z2.h}, p0, x0, #1, mul vl
-    add             x1, x1, #32
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_16_cpy1Dto2D_shl_16x16:
-    ptrue           p0.h, vl16
-.rept 16
-    ld1h            {z1.h}, p0/z, x1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0
-    add             x1, x1, #32
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_32x32_sve)
-    dup             z0.h, w3
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy1Dto2D_shl_32x32
-    ptrue           p0.h, vl8
-.rept 32
-    ld1h            {z1.h}, p0/z, x1
-    ld1h            {z2.h}, p0/z, x1, #1, mul vl
-    ld1h            {z3.h}, p0/z, x1, #2, mul vl
-    ld1h            {z4.h}, p0/z, x1, #3, mul vl
-    lsl             z1.h, p0/m, z1.h, z0.h
-    lsl             z2.h, p0/m, z2.h, z0.h
-    lsl             z3.h, p0/m, z3.h, z0.h
-    lsl             z4.h, p0/m, z4.h, z0.h
-    st1h            {z1.h}, p0, x0
-    st1h            {z2.h}, p0, x0, #1, mul vl
-    st1h            {z3.h}, p0, x0, #2, mul vl
-    st1h            {z4.h}, p0, x0, #3, mul vl
-    add             x1, x1, #64
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_16_cpy1Dto2D_shl_32x32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_cpy1Dto2D_shl_32x32
-    ptrue           p0.h, vl16
-.rept 32
-    ld1h            {z1.h}, p0/z, x1
-    ld1h            {z2.h}, p0/z, x1, #1, mul vl
-    lsl             z1.h, p0/m, z1.h, z0.h
-    lsl             z2.h, p0/m, z2.h, z0.h
-    st1h            {z1.h}, p0, x0
-    st1h            {z2.h}, p0, x0, #1, mul vl
-    add             x1, x1, #64
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_48_cpy1Dto2D_shl_32x32:
-    ptrue           p0.h, vl32
-.rept 32
-    ld1h            {z1.h}, p0/z, x1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0
-    add             x1, x1, #64
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_64x64_sve)
-    dup             z0.h, w3
-    mov             x8, #64
-    mov             w12, #64
-.L_init_cpy1Dto2D_shl_64x64:
-    sub             w12, w12, 1
-    mov             x9, #0
-    whilelt         p0.h, x9, x8
-.L_cpy1Dto2D_shl_64x64:
-    ld1h            {z1.h}, p0/z, x1, x9, lsl #1
-    lsl             z1.h, p0/m, z1.h, z0.h
-    st1h            {z1.h}, p0, x0, x9, lsl #1
-    inch            x9
-    whilelt         p0.h, x9, x8
-    b.first         .L_cpy1Dto2D_shl_64x64
-    addvl           x1, x1, #1
-    add             x0, x0, x2, lsl #1
-    cbnz            w12, .L_init_cpy1Dto2D_shl_64x64
-    ret
-endfunc
-
-// void cpy1Dto2D_shr(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
-
-function PFX(cpy1Dto2D_shr_16x16_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy1Dto2D_shr_16x16
-    cpy1Dto2D_shr_start
-    mov             w12, #4
-.Loop_cpy1Dto2D_shr_16:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.8h-v3.8h}, x1, #32
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.8h-v3.8h}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shr_16
-    ret
-.vl_gt_16_cpy1Dto2D_shr_16x16:
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    ptrue           p0.h, vl16
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 16
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, #32
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_32x32_sve)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy1Dto2D_shr_32x32
-    cpy1Dto2D_shr_start
-    mov             w12, #16
-.Loop_cpy1Dto2D_shr_32_sve:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sub             v4.8h, v4.8h, v1.8h
-    sub             v5.8h, v5.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shr_32_sve
-    ret
-.vl_gt_16_cpy1Dto2D_shr_32x32:
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    cmp             x9, #48
-    bgt             .vl_gt_48_cpy2Dto1D_shr_32x32
-    ptrue           p0.h, vl16
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 32
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    add             x1, x1, #64
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_48_cpy1Dto2D_shr_32x32:
-    ptrue           p0.h, vl32
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 32
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, #64
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_64x64_sve)
-    dup             z0.h, w3
-    sub             w4, w3, #1
-    dup             z1.h, w4
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_cpy1Dto2D_shr_64x64
-    ptrue           p0.h, vl8
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 128
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    ld1d            {z7.d}, p0/z, x1, #2, mul vl
-    ld1d            {z8.d}, p0/z, x1, #3, mul vl
-    ld1d            {z9.d}, p0/z, x1, #4, mul vl
-    ld1d            {z10.d}, p0/z, x1, #5, mul vl
-    ld1d            {z11.d}, p0/z, x1, #6, mul vl
-    ld1d            {z12.d}, p0/z, x1, #7, mul vl
-    add             x1, x1, #128
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    add             z7.h, p0/m, z7.h, z2.h
-    add             z8.h, p0/m, z8.h, z2.h
-    add             z9.h, p0/m, z9.h, z2.h
-    add             z10.h, p0/m, z10.h, z2.h
-    add             z11.h, p0/m, z11.h, z2.h
-    add             z12.h, p0/m, z12.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    asr             z7.h, p0/m, z7.h, z0.h
-    asr             z8.h, p0/m, z8.h, z0.h
-    asr             z9.h, p0/m, z9.h, z0.h
-    asr             z10.h, p0/m, z10.h, z0.h
-    asr             z11.h, p0/m, z11.h, z0.h
-    asr             z12.h, p0/m, z12.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    st1d            {z7.d}, p0, x0, #2, mul vl
-    st1d            {z8.d}, p0, x0, #3, mul vl
-    st1d            {z9.d}, p0, x0, #4, mul vl
-    st1d            {z10.d}, p0, x0, #5, mul vl
-    st1d            {z11.d}, p0, x0, #6, mul vl
-    st1d            {z12.d}, p0, x0, #7, mul vl
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_16_cpy1Dto2D_shr_64x64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_cpy1Dto2D_shr_64x64
-    ptrue           p0.h, vl16
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 128
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    ld1d            {z7.d}, p0/z, x1, #2, mul vl
-    ld1d            {z8.d}, p0/z, x1, #3, mul vl
-    add             x1, x1, #128
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    add             z7.h, p0/m, z7.h, z2.h
-    add             z8.h, p0/m, z8.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    asr             z7.h, p0/m, z7.h, z0.h
-    asr             z8.h, p0/m, z8.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    st1d            {z7.d}, p0, x0, #2, mul vl
-    st1d            {z8.d}, p0, x0, #3, mul vl
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_48_cpy1Dto2D_shr_64x64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_cpy1Dto2D_shr_64x64
-    ptrue           p0.h, vl32
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 128
-    ld1d            {z5.d}, p0/z, x1
-    ld1d            {z6.d}, p0/z, x1, #1, mul vl
-    add             x1, x1, #128
-    add             z5.h, p0/m, z5.h, z2.h
-    add             z6.h, p0/m, z6.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    asr             z6.h, p0/m, z6.h, z0.h
-    st1d            {z5.d}, p0, x0
-    st1d            {z6.d}, p0, x0, #1, mul vl
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-.vl_gt_112_cpy1Dto2D_shr_64x64:
-    ptrue           p0.h, vl64
-    mov             z2.h, #1
-    lsl             z2.h, p0/m, z2.h, z1.h
-.rept 128
-    ld1d            {z5.d}, p0/z, x1
-    add             x1, x1, #128
-    add             z5.h, p0/m, z5.h, z2.h
-    asr             z5.h, p0/m, z5.h, z0.h
-    st1d            {z5.d}, p0, x0
-    add             x0, x0, x2, lsl #1
-.endr
-    ret
-endfunc

x265-4.1.tar/source/common/aarch64/ssd-a-sve2.S Deleted

@@ -1,626 +0,0 @@
-/*****************************************************************************
- * Copyright (C) 2022-2023 MulticoreWare, Inc
- *
- * Authors: David Chen <david.chen@myais.com.cn>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
- *
- * This program is also available under a commercial proprietary license.
- * For more information, contact us at license @ x265.com.
- *****************************************************************************/
-
-#include "asm-sve.S"
-#include "ssd-a-common.S"
-
-.arch armv8-a+sve2
-
-#ifdef __APPLE__
-.section __RODATA,__rodata
-#else
-.section .rodata
-#endif
-
-.align 4
-
-.text
-
-function PFX(pixel_sse_ss_4x4_sve2)
-    ptrue           p0.b, vl8
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z17.h
-    smullb          z3.s, z1.h, z1.h
-    smullt          z4.s, z1.h, z1.h
-.rept 3
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z17.h
-    smlalb          z3.s, z1.h, z1.h
-    smlalt          z4.s, z1.h, z1.h
-.endr
-    uaddv           d3, p0, z3.s
-    fmov            w0, s3
-    uaddv           d4, p0, z4.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-endfunc
-
-function PFX(pixel_sse_ss_8x8_sve2)
-    ptrue           p0.b, vl16
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z17.h
-    smullb          z3.s, z1.h, z1.h
-    smullt          z4.s, z1.h, z1.h
-.rept 7
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z17.h
-    smlalb          z3.s, z1.h, z1.h
-    smlalt          z4.s, z1.h, z1.h
-.endr
-    uaddv           d3, p0, z3.s
-    fmov            w0, s3
-    uaddv           d4, p0, z4.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-endfunc
-
-function PFX(pixel_sse_ss_16x16_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_sse_ss_16x16
-    ptrue           p0.b, vl16
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z18.b}, p0/z, x2
-    ld1b            {z19.b}, p0/z, x2, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z18.h
-    sub             z2.h, z17.h, z19.h
-    smullb          z3.s, z1.h, z1.h
-    smullt          z4.s, z1.h, z1.h
-    smlalb          z3.s, z2.h, z2.h
-    smlalt          z4.s, z2.h, z2.h
-.rept 15
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z18.b}, p0/z, x2
-    ld1b            {z19.b}, p0/z, x2, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z18.h
-    sub             z2.h, z17.h, z19.h
-    smlalb          z3.s, z1.h, z1.h
-    smlalt          z4.s, z1.h, z1.h
-    smlalb          z3.s, z2.h, z2.h
-    smlalt          z4.s, z2.h, z2.h
-.endr
-    uaddv           d3, p0, z3.s
-    fmov            w0, s3
-    uaddv           d4, p0, z4.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_16_pixel_sse_ss_16x16:
-    ptrue           p0.b, vl32
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z18.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z18.h
-    smullb          z3.s, z1.h, z1.h
-    smullt          z4.s, z1.h, z1.h
-.rept 15
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z18.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z18.h
-    smlalb          z3.s, z1.h, z1.h
-    smlalt          z4.s, z1.h, z1.h
-.endr
-    uaddv           d3, p0, z3.s
-    fmov            w0, s3
-    uaddv           d4, p0, z4.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-endfunc
-
-function PFX(pixel_sse_ss_32x32_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_sse_ss_32x32
-    ptrue           p0.b, vl16
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z18.b}, p0/z, x0, #2, mul vl
-    ld1b            {z19.b}, p0/z, x0, #3, mul vl
-    ld1b            {z20.b}, p0/z, x2
-    ld1b            {z21.b}, p0/z, x2, #1, mul vl
-    ld1b            {z22.b}, p0/z, x2, #2, mul vl
-    ld1b            {z23.b}, p0/z, x2, #3, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    sub             z2.h, z17.h, z21.h
-    sub             z3.h, z18.h, z22.h
-    sub             z4.h, z19.h, z23.h
-    smullb          z5.s, z1.h, z1.h
-    smullt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    smlalb          z5.s, z4.h, z4.h
-    smlalt          z6.s, z4.h, z4.h
-.rept 31
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z18.b}, p0/z, x0, #2, mul vl
-    ld1b            {z19.b}, p0/z, x0, #3, mul vl
-    ld1b            {z20.b}, p0/z, x2
-    ld1b            {z21.b}, p0/z, x2, #1, mul vl
-    ld1b            {z22.b}, p0/z, x2, #2, mul vl
-    ld1b            {z23.b}, p0/z, x2, #3, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    sub             z2.h, z17.h, z21.h
-    sub             z3.h, z18.h, z22.h
-    sub             z4.h, z19.h, z23.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    smlalb          z5.s, z4.h, z4.h
-    smlalt          z6.s, z4.h, z4.h
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_16_pixel_sse_ss_32x32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_sse_ss_32x32
-    ptrue           p0.b, vl32
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z20.b}, p0/z, x2
-    ld1b            {z21.b}, p0/z, x2, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    sub             z2.h, z17.h, z21.h
-    smullb          z5.s, z1.h, z1.h
-    smullt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-.rept 31
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    ld1b            {z20.b}, p0/z, x2
-    ld1b            {z21.b}, p0/z, x2, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    sub             z2.h, z17.h, z21.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_48_pixel_sse_ss_32x32:
-    ptrue           p0.b, vl64
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z20.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    smullb          z5.s, z1.h, z1.h
-    smullt          z6.s, z1.h, z1.h
-.rept 31
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z20.b}, p0/z, x2
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-    sub             z1.h, z16.h, z20.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-endfunc
-
-function PFX(pixel_sse_ss_64x64_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_sse_ss_64x64
-    ptrue           p0.b, vl16
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z25.b}, p0/z, x0, #1, mul vl
-    ld1b            {z26.b}, p0/z, x0, #2, mul vl
-    ld1b            {z27.b}, p0/z, x0, #3, mul vl
-    ld1b            {z28.b}, p0/z, x2
-    ld1b            {z29.b}, p0/z, x2, #1, mul vl
-    ld1b            {z30.b}, p0/z, x2, #2, mul vl
-    ld1b            {z31.b}, p0/z, x2, #3, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    sub             z2.h, z26.h, z30.h
-    sub             z3.h, z27.h, z31.h
-    smullb          z5.s, z0.h, z0.h
-    smullt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    ld1b            {z24.b}, p0/z, x0, #4, mul vl
-    ld1b            {z25.b}, p0/z, x0, #5, mul vl
-    ld1b            {z26.b}, p0/z, x0, #6, mul vl
-    ld1b            {z27.b}, p0/z, x0, #7, mul vl
-    ld1b            {z28.b}, p0/z, x2, #4, mul vl
-    ld1b            {z29.b}, p0/z, x2, #5, mul vl
-    ld1b            {z30.b}, p0/z, x2, #6, mul vl
-    ld1b            {z31.b}, p0/z, x2, #7, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    sub             z2.h, z26.h, z30.h
-    sub             z3.h, z27.h, z31.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.rept 63
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z25.b}, p0/z, x0, #1, mul vl
-    ld1b            {z26.b}, p0/z, x0, #2, mul vl
-    ld1b            {z27.b}, p0/z, x0, #3, mul vl
-    ld1b            {z28.b}, p0/z, x2
-    ld1b            {z29.b}, p0/z, x2, #1, mul vl
-    ld1b            {z30.b}, p0/z, x2, #2, mul vl
-    ld1b            {z31.b}, p0/z, x2, #3, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    sub             z2.h, z26.h, z30.h
-    sub             z3.h, z27.h, z31.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    ld1b            {z24.b}, p0/z, x0, #4, mul vl
-    ld1b            {z25.b}, p0/z, x0, #5, mul vl
-    ld1b            {z26.b}, p0/z, x0, #6, mul vl
-    ld1b            {z27.b}, p0/z, x0, #7, mul vl
-    ld1b            {z28.b}, p0/z, x2, #4, mul vl
-    ld1b            {z29.b}, p0/z, x2, #5, mul vl
-    ld1b            {z30.b}, p0/z, x2, #6, mul vl
-    ld1b            {z31.b}, p0/z, x2, #7, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    sub             z2.h, z26.h, z30.h
-    sub             z3.h, z27.h, z31.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    smlalb          z5.s, z2.h, z2.h
-    smlalt          z6.s, z2.h, z2.h
-    smlalb          z5.s, z3.h, z3.h
-    smlalt          z6.s, z3.h, z3.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_16_pixel_sse_ss_64x64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_sse_ss_64x64
-    ptrue           p0.b, vl32
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z25.b}, p0/z, x0, #1, mul vl
-    ld1b            {z28.b}, p0/z, x2
-    ld1b            {z29.b}, p0/z, x2, #1, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    smullb          z5.s, z0.h, z0.h
-    smullt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    ld1b            {z24.b}, p0/z, x0, #1, mul vl
-    ld1b            {z25.b}, p0/z, x0, #2, mul vl
-    ld1b            {z28.b}, p0/z, x2, #1, mul vl
-    ld1b            {z29.b}, p0/z, x2, #2, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.rept 63
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z25.b}, p0/z, x0, #1, mul vl
-    ld1b            {z28.b}, p0/z, x2
-    ld1b            {z29.b}, p0/z, x2, #1, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    ld1b            {z24.b}, p0/z, x0, #1, mul vl
-    ld1b            {z25.b}, p0/z, x0, #2, mul vl
-    ld1b            {z28.b}, p0/z, x2, #1, mul vl
-    ld1b            {z29.b}, p0/z, x2, #2, mul vl
-    sub             z0.h, z24.h, z28.h
-    sub             z1.h, z25.h, z29.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    smlalb          z5.s, z1.h, z1.h
-    smlalt          z6.s, z1.h, z1.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_48_pixel_sse_ss_64x64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_pixel_sse_ss_64x64
-    ptrue           p0.b, vl64
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z28.b}, p0/z, x2
-    sub             z0.h, z24.h, z28.h
-    smullb          z5.s, z0.h, z0.h
-    smullt          z6.s, z0.h, z0.h
-    ld1b            {z24.b}, p0/z, x0, #1, mul vl
-    ld1b            {z28.b}, p0/z, x2, #1, mul vl
-    sub             z0.h, z24.h, z28.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.rept 63
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z28.b}, p0/z, x2
-    sub             z0.h, z24.h, z28.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    ld1b            {z24.b}, p0/z, x0, #1, mul vl
-    ld1b            {z28.b}, p0/z, x2, #1, mul vl
-    sub             z0.h, z24.h, z28.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-.vl_gt_112_pixel_sse_ss_64x64:
-    ptrue           p0.b, vl128
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z28.b}, p0/z, x2
-    sub             z0.h, z24.h, z28.h
-    smullb          z5.s, z0.h, z0.h
-    smullt          z6.s, z0.h, z0.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.rept 63
-    ld1b            {z24.b}, p0/z, x0
-    ld1b            {z28.b}, p0/z, x2
-    sub             z0.h, z24.h, z28.h
-    smlalb          z5.s, z0.h, z0.h
-    smlalt          z6.s, z0.h, z0.h
-    add             x0, x0, x1, lsl #1
-    add             x2, x2, x3, lsl #1
-.endr
-    uaddv           d3, p0, z5.s
-    fmov            w0, s3
-    uaddv           d4, p0, z6.s
-    fmov            w1, s4
-    add             w0, w0, w1
-    ret
-endfunc
-
-function PFX(pixel_ssd_s_4x4_sve2)
-    ptrue           p0.b, vl8
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smullb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.rept 3
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smlalb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.endr
-    uaddv           d3, p0, z0.s
-    fmov            w0, s3
-    ret
-endfunc
-
-function PFX(pixel_ssd_s_8x8_sve2)
-    ptrue           p0.b, vl16
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smullb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.rept 7
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smlalb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.endr
-    uaddv           d3, p0, z0.s
-    fmov            w0, s3
-    ret
-endfunc
-
-function PFX(pixel_ssd_s_16x16_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_ssd_s_16x16
-    add             x1, x1, x1
-    mov             w12, #4
-    movi            v0.16b, #0
-    movi            v1.16b, #0
-.Loop_ssd_s_16_sve2:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v4.16b,v5.16b}, x0, x1
-    ld1             {v6.16b,v7.16b}, x0, x1
-    smlal           v0.4s, v4.4h, v4.4h
-    smlal2          v1.4s, v4.8h, v4.8h
-    smlal           v0.4s, v5.4h, v5.4h
-    smlal2          v1.4s, v5.8h, v5.8h
-    smlal           v0.4s, v6.4h, v6.4h
-    smlal2          v1.4s, v6.8h, v6.8h
-    smlal           v0.4s, v7.4h, v7.4h
-    smlal2          v1.4s, v7.8h, v7.8h
-.endr
-    cbnz            w12, .Loop_ssd_s_16_sve2
-    add             v0.4s, v0.4s, v1.4s
-    ret_v0_w0
-.vl_gt_16_pixel_ssd_s_16x16:
-    ptrue           p0.b, vl32
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smullb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.rept 15
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smlalb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.endr
-    uaddv           d3, p0, z0.s
-    fmov            w0, s3
-    ret
-endfunc
-
-function PFX(pixel_ssd_s_32x32_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_ssd_s_32x32
-    add             x1, x1, x1
-    mov             w12, #8
-    movi            v0.16b, #0
-    movi            v1.16b, #0
-.Loop_ssd_s_32:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v4.16b-v7.16b}, x0, x1
-    smlal           v0.4s, v4.4h, v4.4h
-    smlal2          v1.4s, v4.8h, v4.8h
-    smlal           v0.4s, v5.4h, v5.4h
-    smlal2          v1.4s, v5.8h, v5.8h
-    smlal           v0.4s, v6.4h, v6.4h
-    smlal2          v1.4s, v6.8h, v6.8h
-    smlal           v0.4s, v7.4h, v7.4h
-    smlal2          v1.4s, v7.8h, v7.8h
-.endr
-    cbnz            w12, .Loop_ssd_s_32
-    add             v0.4s, v0.4s, v1.4s
-    ret_v0_w0
-.vl_gt_16_pixel_ssd_s_32x32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_ssd_s_32x32
-    ptrue           p0.b, vl32
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    smullb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-    smlalb          z0.s, z17.h, z17.h
-    smlalt          z0.s, z17.h, z17.h
-.rept 31
-    ld1b            {z16.b}, p0/z, x0
-    ld1b            {z17.b}, p0/z, x0, #1, mul vl
-    add             x0, x0, x1, lsl #1
-    smlalb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-    smlalb          z0.s, z17.h, z17.h
-    smlalt          z0.s, z17.h, z17.h
-.endr
-    uaddv           d3, p0, z0.s
-    fmov            w0, s3
-    ret
-.vl_gt_48_pixel_ssd_s_32x32:
-    ptrue           p0.b, vl64
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smullb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.rept 31
-    ld1b            {z16.b}, p0/z, x0
-    add             x0, x0, x1, lsl #1
-    smlalb          z0.s, z16.h, z16.h
-    smlalt          z0.s, z16.h, z16.h
-.endr
-    uaddv           d3, p0, z0.s
-    fmov            w0, s3
-    ret
-endfunc

x265-4.2.tar/bitbucket-pipelines.yml Added

@@ -0,0 +1,716 @@
+image: ubuntu:22.04
+
+definitions:
+  caches:
+    ccache: ~/.ccache
+
+  steps:
+    - step: &build-8bit
+        name: "Build 8-bit"
+        caches:
+          - ccache
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm ccache libnuma-dev git > /dev/null 2>&1
+          - export PATH="/usr/lib/ccache:$PATH"
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_SHARED=ON -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON -DENABLE_TESTS=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - echo "--- Verify build artifacts ---"
+          - test -f x265 && echo "CLI binary OK"
+          - test -f libx265.so && echo "Shared library OK"
+          - test -f libx265.a && echo "Static library OK"
+          - ./x265 --version
+          - echo "--- Run TestBench (unit tests) ---"
+          - ./test/TestBench || (echo "TestBench FAILED"; exit 1)
+        artifacts:
+          - build/8bit/x265
+          - build/8bit/libx265.so*
+          - build/8bit/libx265.a
+
+    - step: &build-10bit
+        name: "Build 10-bit"
+        caches:
+          - ccache
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm ccache libnuma-dev git > /dev/null 2>&1
+          - export PATH="/usr/lib/ccache:$PATH"
+          - mkdir -p build/10bit && cd build/10bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DENABLE_SHARED=OFF -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON -DENABLE_TESTS=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - test -f x265 && echo "10-bit CLI binary OK"
+          - ./x265 --version
+          - ./test/TestBench || (echo "10-bit TestBench FAILED"; exit 1)
+        artifacts:
+          - build/10bit/libx265.a
+
+    - step: &build-12bit
+        name: "Build 12-bit"
+        caches:
+          - ccache
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm ccache libnuma-dev git > /dev/null 2>&1
+          - export PATH="/usr/lib/ccache:$PATH"
+          - mkdir -p build/12bit && cd build/12bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DMAIN12=ON -DENABLE_SHARED=OFF -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON -DENABLE_TESTS=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - test -f x265 && echo "12-bit CLI binary OK"
+          - ./x265 --version
+          - ./test/TestBench || (echo "12-bit TestBench FAILED"; exit 1)
+        artifacts:
+          - build/12bit/libx265.a
+
+    - step: &build-multilib
+        name: "Build Multi-lib (8+10+12 bit)"
+        caches:
+          - ccache
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm ccache libnuma-dev git > /dev/null 2>&1
+          - export PATH="/usr/lib/ccache:$PATH"
+          - mkdir -p build/multilib
+          - mkdir -p build/multilib/12bit && cd build/multilib/12bit
+          - cmake ../../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DMAIN12=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DEXPORT_C_API=OFF -DENABLE_ASSEMBLY=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - cd ../../..
+          - mkdir -p build/multilib/10bit && cd build/multilib/10bit
+          - cmake ../../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DEXPORT_C_API=OFF -DENABLE_ASSEMBLY=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - cd ../../..
+          - mkdir -p build/multilib/8bit && cd build/multilib/8bit
+          - ln -sf ../../multilib/10bit/libx265.a libx265_main10.a
+          - ln -sf ../../multilib/12bit/libx265.a libx265_main12.a
+          - cmake ../../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_SHARED=ON -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON -DEXTRA_LIB="x265_main10.a;x265_main12.a" -DEXTRA_LINK_FLAGS="-L." -DLINKED_10BIT=ON -DLINKED_12BIT=ON -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
+          - make -j$(nproc)
+          - test -f x265 && echo "Multi-lib CLI binary OK"
+          - ./x265 --version
+        artifacts:
+          - build/multilib/8bit/x265
+          - build/multilib/8bit/libx265.so*
+          - build/multilib/8bit/libx265.a
+
+    - step: &test-cli-presets
+        name: "Test CLI - All Presets & Tune Modes"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p testdata/test_420_416x240.yuv
+          - X265=build/8bit/x265
+          - PASS=0
+          - FAIL=0
+          - TOTAL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - |
+            for preset in ultrafast superfast veryfast faster fast medium slow slower veryslow placebo; do
+              TOTAL=$((TOTAL + 1))
+              if run_x265 "Preset $preset" $X265 --preset $preset --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out_${preset}.hevc testdata/test_420_416x240.yuv; then
+                PASS=$((PASS + 1))
+              else
+                FAIL=$((FAIL + 1))
+              fi
+              rm -f /tmp/out_${preset}.hevc
+            done
+          - |
+            for tune in psnr ssim grain fastdecode zerolatency; do
+              TOTAL=$((TOTAL + 1))
+              if run_x265 "Tune $tune" $X265 --preset medium --tune $tune --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out_tune_${tune}.hevc testdata/test_420_416x240.yuv; then
+                PASS=$((PASS + 1))
+              else
+                FAIL=$((FAIL + 1))
+              fi
+              rm -f /tmp/out_tune_${tune}.hevc
+            done
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Preset/Tune Results $PASS/$TOTAL passed, $FAIL failed ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/cli-presets.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-rate-control
+        name: "Test Rate Control Modes"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=3:size=832x480:rate=30 -pix_fmt yuv420p testdata/test_420_832x480.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "CRF Mode" $X265 --preset medium --crf 22 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_crf.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "CQP Mode" $X265 --preset medium --qp 28 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_cqp.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "ABR Mode" $X265 --preset medium --bitrate 1000 -F4 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_abr.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "VBV Constrained" $X265 --preset medium --bitrate 1000 --vbv-maxrate 1000 --vbv-bufsize 1000 -F4 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_vbv.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Strict CBR" $X265 --preset medium --bitrate 1000 --vbv-maxrate 1000 --vbv-bufsize 1000 --strict-cbr -F4 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_cbr.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "CRF + VBV" $X265 --preset medium --crf 22 --vbv-maxrate 2000 --vbv-bufsize 3000 --crf-max 32 --crf-min 18 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_crf_vbv.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Two-pass ABR pass1" $X265 --preset medium --bitrate 1000 --pass 1 -F4 --input-res 832x480 --fps 30 --frames 60 -o /dev/null testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Two-pass ABR pass2" $X265 --preset medium --bitrate 1000 --pass 2 -F4 --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_2pass.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Multi-pass Opt Analysis pass1" $X265 --preset medium --bitrate 1000 --pass 1 --multi-pass-opt-analysis --input-res 832x480 --fps 30 --frames 60 -o /dev/null testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Multi-pass Opt Analysis pass2" $X265 --preset medium --bitrate 1000 --pass 2 --multi-pass-opt-analysis --ssim --psnr --input-res 832x480 --fps 30 --frames 60 -o /tmp/out_mp_opt.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out_*.hevc x265_2pass.log x265_2pass.log.cutree
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Rate Control Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/rate-control.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-encoding-features
+        name: "Test Encoding Features"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=832x480:rate=30 -pix_fmt yuv420p testdata/test_420_832x480.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv422p testdata/test_422_416x240.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv444p testdata/test_444_416x240.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p testdata/test_420_416x240.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "Lossless" $X265 --preset ultrafast --lossless --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "CU Lossless" $X265 --preset medium --crf 22 --cu-lossless --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - |
+            for ctu in 16 32 64; do
+              run_x265 "CTU $ctu" $X265 --preset medium --ctu $ctu --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - |
+            for mcu in 8 16 32; do
+              run_x265 "Min-CU $mcu" $X265 --preset medium --min-cu-size $mcu --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - |
+            for aqmode in 0 1 2 3 4; do
+              run_x265 "AQ Mode $aqmode" $X265 --preset medium --aq-mode $aqmode --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - |
+            for me in dia hex umh star sea full; do
+              run_x265 "ME $me" $X265 --preset medium --me $me --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - run_x265 "SAO + Deblock" $X265 --preset medium --sao --deblock 2:2 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "No SAO + No Deblock" $X265 --preset medium --no-sao --no-deblock --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Limit SAO" $X265 --preset medium --limit-sao --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Weighted P + B" $X265 --preset medium --weightp --weightb --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "B-frames 0" $X265 --preset medium --bframes 0 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "B-frames 16" $X265 --preset medium --bframes 16 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Infinite Keyint" $X265 --preset medium --keyint -1 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Short Keyint" $X265 --preset medium --keyint 10 --min-keyint 5 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Closed GOP" $X265 --preset medium --no-open-gop --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Constrained Intra" $X265 --preset medium --constrained-intra --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - |
+            for rd in 1 2 3 4 5 6; do
+              run_x265 "RD Level $rd" $X265 --preset medium --rd $rd --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - |
+            for rdoq in 0 1 2; do
+              run_x265 "RDOQ Level $rdoq" $X265 --preset medium --rdoq-level $rdoq --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+            done
+          - run_x265 "Scaling List Default" $X265 --preset medium --scaling-list default --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Noise Reduction" $X265 --preset medium --nr-intra 500 --nr-inter 300 -F4 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Psy-RD + Psy-RDOQ" $X265 --preset medium --psy-rd 1.5 --psy-rdoq 1.0 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "No Psy" $X265 --preset medium --no-psy-rd --no-psy-rdoq --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Multiple Slices" $X265 --preset medium --slices 4 --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Temporal Layers" $X265 --preset medium --temporal-layers 3 --b-adapt 0 --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "HEVC-AQ" $X265 --preset medium --hevc-aq --no-cutree --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "HME" $X265 --preset medium --hme --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Lowpass DCT" $X265 --preset medium --lowpass-dct --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Streaming Flags" $X265 --preset medium --hrd --aud --repeat-headers --bitrate 1000 --vbv-maxrate 1000 --vbv-bufsize 1000 -F4 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "422 Input" $X265 --preset medium --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_422_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "444 Input" $X265 --preset medium --crf 28 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_444_416x240.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Encoding Features Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/encoding-features.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-analysis-save-load
+        name: "Test Analysis Save/Load & Segment Encoding"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=4:size=416x240:rate=30 -pix_fmt yuv420p testdata/test_420_416x240.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "Analysis Save (level 2)" $X265 --preset medium --analysis-save x265_analysis.dat --analysis-save-reuse-level 2 --bitrate 500 --input-res 416x240 --fps 30 --frames 60 -o /tmp/out_save.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Analysis Load (level 2)" $X265 --preset medium --analysis-load x265_analysis.dat --analysis-load-reuse-level 2 --bitrate 500 --input-res 416x240 --fps 30 --frames 60 -o /tmp/out_load.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Analysis Save (level 10)" $X265 --preset slow --analysis-save x265_analysis_10.dat --analysis-save-reuse-level 10 --bitrate 500 --input-res 416x240 --fps 30 --frames 60 -o /tmp/out_save10.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Analysis Load (level 10)" $X265 --preset slow --analysis-load x265_analysis_10.dat --analysis-load-reuse-level 10 --bitrate 500 --input-res 416x240 --fps 30 --frames 60 -o /tmp/out_load10.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Segment Encoding" $X265 --preset ultrafast --no-open-gop --chunk-start 10 --chunk-end 50 --crf 28 --input-res 416x240 --fps 30 --frames 60 -o /tmp/out_seg.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out_*.hevc x265_analysis*.dat
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Analysis/Segment Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/analysis-save-load.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-bitstream-validation
+        name: "Test Bitstream Validation (decode output)"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p testdata/test_420_416x240.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            echo "" > /tmp/summary.log
+            validate_encode() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              local outfile="/tmp/validate_${RANDOM}.hevc"
+              $X265 "$@" -o "$outfile" testdata/test_420_416x240.yuv 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              if  $rc -eq 0 ; then
+                if ffprobe -v error -show_entries stream=codec_name "$outfile" 2>&1 | grep -q hevc; then
+                  echo "PASS: $label (encode + probe)"
+                  if ffmpeg -y -i "$outfile" -frames:v 5 -f rawvideo /dev/null 2>&1; then
+                    echo "PASS: $label (decode)"
+                  else
+                    echo "FAIL: $label (decode failed)"
+                    FAIL=$((FAIL + 1))
+                  fi
+                else
+                  echo "FAIL: $label (not valid HEVC)"
+                  FAIL=$((FAIL + 1))
+                fi
+              else
+                echo "FAIL: $label (encode failed)"
+                FAIL=$((FAIL + 1))
+              fi
+              rm -f "$outfile"
+            }
+          - validate_encode "CRF ultrafast" --preset ultrafast --crf 28 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "CRF medium" --preset medium --crf 22 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "CRF veryslow" --preset veryslow --crf 18 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "ABR + VBV" --preset medium --bitrate 500 --vbv-maxrate 500 --vbv-bufsize 500 -F4 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "Lossless" --preset ultrafast --lossless --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "HRD compliant" --preset medium --bitrate 500 --vbv-maxrate 500 --vbv-bufsize 500 --hrd --aud --repeat-headers -F4 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "Closed GOP" --preset medium --no-open-gop --keyint 10 --crf 28 --input-res 416x240 --fps 30 --frames 30
+          - validate_encode "Zero-latency" --preset medium --tune zerolatency --crf 28 --input-res 416x240 --fps 30 --frames 30
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Bitstream Validation Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/bitstream-validation.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-threading
+        name: "Test Threading & Parallelism"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=832x480:rate=30 -pix_fmt yuv420p testdata/test_420_832x480.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "WPP enabled" $X265 --preset medium --wpp --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "WPP disabled" $X265 --preset medium --no-wpp --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - |
+            for ft in 1 2 4; do
+              run_x265 "Frame threads $ft" $X265 --preset medium --frame-threads $ft --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+            done
+          - |
+            for ls in 0 2 4; do
+              run_x265 "Lookahead slices $ls" $X265 --preset medium --lookahead-slices $ls --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+            done
+          - run_x265 "Pools=none (single thread)" $X265 --preset ultrafast --pools none --crf 28 --input-res 832x480 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Threading Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/threading.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-quality-metrics
+        name: "Test Quality Metrics (SSIM/PSNR)"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p testdata/test_420_416x240.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "SSIM Metric" $X265 --preset medium --ssim --crf 22 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - grep -qi "ssim" /tmp/x265_last.log || FAIL=$((FAIL + 1))
+          - run_x265 "PSNR Metric" $X265 --preset medium --psnr --crf 22 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - grep -qi "psnr" /tmp/x265_last.log || FAIL=$((FAIL + 1))
+          - run_x265 "SSIM + PSNR" $X265 --preset medium --ssim --psnr --crf 22 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - grep -qi "ssim" /tmp/x265_last.log || FAIL=$((FAIL + 1))
+          - grep -qi "psnr" /tmp/x265_last.log || FAIL=$((FAIL + 1))
+          - run_x265 "Hash (MD5)" $X265 --preset medium --hash 1 --crf 22 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Recon Output" $X265 --preset medium --crf 22 --input-res 416x240 --fps 30 --frames 30 -o /tmp/out.hevc -r /tmp/recon.yuv testdata/test_420_416x240.yuv || FAIL=$((FAIL + 1))
+          - test -f /tmp/recon.yuv && echo "Recon file created OK" || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc /tmp/recon.yuv
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Quality Metrics Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/quality-metrics.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-realworld-video
+        name: "Test Real-World Video (Y4M + Artifacts)"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg wget > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata artifacts
+          - echo "=== Downloading xiph.org test sequences ==="
+          - wget -q -O testdata/bus_cif.y4m https://media.xiph.org/video/derf/y4m/bus_cif.y4m
+          - wget -q -O testdata/akiyo_cif.y4m https://media.xiph.org/video/derf/y4m/akiyo_cif.y4m
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "bus ultrafast CRF 28" $X265 --preset ultrafast --crf 28 --ssim --psnr --frames 150 -o artifacts/bus_ultrafast_crf28.hevc testdata/bus_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "bus medium CRF 22" $X265 --preset medium --crf 22 --ssim --psnr --frames 150 -o artifacts/bus_medium_crf22.hevc testdata/bus_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "bus slow ABR 1000k" $X265 --preset slow --bitrate 1000 --vbv-maxrate 1500 --vbv-bufsize 2000 -F4 --ssim --psnr --frames 150 -o artifacts/bus_slow_abr1000k.hevc testdata/bus_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "bus veryslow tune grain" $X265 --preset veryslow --tune grain --crf 20 --ssim --psnr --frames 150 -o artifacts/bus_veryslow_grain.hevc testdata/bus_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "akiyo medium CRF 22" $X265 --preset medium --crf 22 --ssim --psnr --frames 300 -o artifacts/akiyo_medium_crf22.hevc testdata/akiyo_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "akiyo slow VBV 500k CBR" $X265 --preset slow --bitrate 500 --vbv-maxrate 500 --vbv-bufsize 500 --strict-cbr -F4 --ssim --psnr --frames 300 -o artifacts/akiyo_slow_vbv500k.hevc testdata/akiyo_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "akiyo veryslow CRF 18" $X265 --preset veryslow --crf 18 --ssim --psnr --frames 300 -o artifacts/akiyo_veryslow_crf18.hevc testdata/akiyo_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "akiyo 2-pass pass1" $X265 --preset medium --bitrate 400 --pass 1 -F4 --frames 300 -o /dev/null testdata/akiyo_cif.y4m || FAIL=$((FAIL + 1))
+          - run_x265 "akiyo 2-pass pass2" $X265 --preset medium --bitrate 400 --pass 2 -F4 --ssim --psnr --frames 300 -o artifacts/akiyo_medium_2pass_400k.hevc testdata/akiyo_cif.y4m || FAIL=$((FAIL + 1))
+          - |
+            echo "=== Validate all artifacts with ffprobe ==="
+            for f in artifacts/*.hevc; do
+              if ffprobe -v error -show_entries stream=codec_name "$f" 2>&1 | grep -q hevc; then
+                echo "PASS: $f is valid HEVC"
+              else
+                echo "FAIL: $f is NOT valid HEVC"
+                FAIL=$((FAIL + 1))
+              fi
+            done
+          - rm -f x265_2pass.log x265_2pass.log.cutree
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== Real-World Video Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/realworld-video.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - artifacts/*.hevc
+          - summaries/*.log
+
+    - step: &test-10bit-12bit-encoding
+        name: "Test 10-bit & 12-bit Encoding"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=832x480:rate=30 -pix_fmt yuv420p10le testdata/test_10bit_832x480.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p10le testdata/test_10bit_416x240.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=2:size=416x240:rate=30 -pix_fmt yuv420p12le testdata/test_12bit_416x240.yuv
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - echo "=== Building 10-bit CLI ==="
+          - mkdir -p build/10bit && cd build/10bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - X265_10=build/10bit/x265
+          - run_x265 "10-bit ultrafast CRF" $X265_10 --preset ultrafast --crf 28 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit medium CRF" $X265_10 --preset medium --crf 22 --ssim --psnr --input-res 832x480 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit slow ABR + VBV" $X265_10 --preset slow --bitrate 1000 --vbv-maxrate 1000 --vbv-bufsize 1000 -F4 --ssim --psnr --input-res 832x480 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit veryslow CRF" $X265_10 --preset veryslow --crf 18 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit lossless" $X265_10 --preset ultrafast --lossless --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit CQP" $X265_10 --preset medium --qp 28 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit 2-pass pass1" $X265_10 --preset medium --bitrate 500 --pass 1 -F4 --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /dev/null testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit 2-pass pass2" $X265_10 --preset medium --bitrate 500 --pass 2 -F4 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "10-bit tune grain" $X265_10 --preset medium --tune grain --crf 22 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 10 --frames 30 -o /tmp/out.hevc testdata/test_10bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc x265_2pass.log x265_2pass.log.cutree
+          - echo "=== Building 12-bit CLI ==="
+          - mkdir -p build/12bit && cd build/12bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DHIGH_BIT_DEPTH=ON -DMAIN12=ON -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - X265_12=build/12bit/x265
+          - run_x265 "12-bit ultrafast CRF" $X265_12 --preset ultrafast --crf 28 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 12 --frames 30 -o /tmp/out.hevc testdata/test_12bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "12-bit medium CRF" $X265_12 --preset medium --crf 22 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 12 --frames 30 -o /tmp/out.hevc testdata/test_12bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "12-bit slow CRF" $X265_12 --preset slow --crf 20 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 12 --frames 30 -o /tmp/out.hevc testdata/test_12bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "12-bit lossless" $X265_12 --preset ultrafast --lossless --input-res 416x240 --fps 30 --input-depth 12 --frames 30 -o /tmp/out.hevc testdata/test_12bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "12-bit ABR + VBV" $X265_12 --preset medium --bitrate 500 --vbv-maxrate 500 --vbv-bufsize 500 -F4 --ssim --psnr --input-res 416x240 --fps 30 --input-depth 12 --frames 30 -o /tmp/out.hevc testdata/test_12bit_416x240.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== 10-bit/12-bit Encoding Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/10bit-12bit-encoding.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &test-mcstf-sbrc-interlace
+        name: "Test MCSTF, SBRC & Interlace"
+        script:
+          - export DEBIAN_FRONTEND=noninteractive
+          - apt-get update -qq
+          - apt-get install -y -qq build-essential cmake nasm libnuma-dev ffmpeg > /dev/null 2>&1
+          - mkdir -p build/8bit && cd build/8bit
+          - cmake ../../source -DCMAKE_BUILD_TYPE=Release -DENABLE_CLI=ON -DENABLE_ASSEMBLY=ON
+          - make -j$(nproc)
+          - cd ../..
+          - mkdir -p testdata
+          - ffmpeg -y -f lavfi -i testsrc=duration=3:size=832x480:rate=30 -pix_fmt yuv420p testdata/test_420_832x480.yuv
+          - ffmpeg -y -f lavfi -i testsrc=duration=3:size=1920x1080:rate=25 -pix_fmt yuv420p testdata/test_420_1080p.yuv
+          - X265=build/8bit/x265
+          - FAIL=0
+          - |
+            run_x265() {
+              local label="$1"; shift
+              echo "=== $label ==="
+              "$@" 2>&1 | tee /tmp/x265_last.log
+              local rc=${PIPESTATUS0}
+              local summary=$(grep "encoded" /tmp/x265_last.log | tail -1)
+               -n "$summary"  && printf "  %-45s | %s\n" "$label" "$summary" >> /tmp/summary.log
+              return $rc
+            }
+            echo "" > /tmp/summary.log
+          - run_x265 "MCSTF slower" $X265 --preset slower --crf 24 --mcstf --bframes 5 --frame-threads 1 --no-cutree --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "MCSTF medium" $X265 --preset medium --crf 26 --mcstf --bframes 5 --frame-threads 1 --tskip-fast --constrained-intra --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "MCSTF slow" $X265 --preset slow --crf 24 --mcstf --bframes 5 --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "SBRC slow" $X265 --preset slow --crf 26 --sbrc --no-open-gop --keyint 60 --min-keyint 60 --vbv-bufsize 6000 --vbv-maxrate 5000 --temporal-layers 4 --b-adapt 0 --no-cutree --input-res 1920x1080 --fps 25 --frames 50 -o /tmp/out.hevc testdata/test_420_1080p.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "SBRC superfast" $X265 --preset superfast --crf 22 --sbrc --no-open-gop --vbv-maxrate 9000 --vbv-bufsize 7500 --keyint 100 --min-keyint 100 --input-res 1920x1080 --fps 25 --frames 50 -o /tmp/out.hevc testdata/test_420_1080p.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Interlace TFF" $X265 --preset faster --interlace tff --crf 28 --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Interlace BFF" $X265 --preset fast --interlace bff --crf 28 --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - run_x265 "Interlace BFF + weightb" $X265 --preset fast --interlace bff --weightb --crf 28 --input-res 832x480 --fps 30 --frames 60 -o /tmp/out.hevc testdata/test_420_832x480.yuv || FAIL=$((FAIL + 1))
+          - rm -f /tmp/out.hevc
+          - echo ""
+          - echo "==================== ENCODING SUMMARY ===================="
+          - cat /tmp/summary.log 2>/dev/null || true
+          - echo "=========================================================="
+          - echo "=== MCSTF/SBRC/Interlace Results FAIL=$FAIL ==="
+          - mkdir -p summaries && cp /tmp/summary.log summaries/mcstf-sbrc-interlace.log 2>/dev/null || true
+          - test $FAIL -eq 0
+        artifacts:
+          - summaries/*.log
+
+    - step: &summary-report
+        name: "Pipeline Summary Report"
+        script:
+          - echo ""
+          - echo "================================================================"
+          - echo "       x265 PIPELINE - COMBINED ENCODING SUMMARY REPORT"
+          - echo "================================================================"
+          - |
+            total_encodes=0
+            for f in summaries/*.log; do
+               -f "$f"  || continue
+              step_name=$(basename "$f" .log)
+              count=$(grep -c "encoded" "$f" 2>/dev/null || echo 0)
+              total_encodes=$((total_encodes + count))
+              echo ""
+              echo "----------------------------------------------------------------"
+              printf " %s (%d encodes)\n" "$step_name" "$count"
+              echo "----------------------------------------------------------------"
+              echo "  TEST                                          | FRAMES  FPS       BITRATE     QP"
+              echo "  ----------------------------------------------|--------------------------------------"
+              while IFS= read -r line; do
+                if  -n "$line" ; then
+                  echo "$line"
+                fi
+              done < "$f"
+            done
+          - echo ""
+          - echo "================================================================"
+          - |
+            total_encodes=0
+            total_fps=0
+            fps_count=0
+            for f in summaries/*.log; do
+               -f "$f"  || continue
+              while IFS= read -r line; do
+                fps=$(echo "$line" | grep -oP '$\d.+\s+fps$' | grep -oP '\d.+' | head -1)
+                if  -n "$fps" ; then
+                  total_encodes=$((total_encodes + 1))
+                  total_fps=$(echo "$total_fps + $fps" | bc 2>/dev/null || echo "$total_fps")
+                  fps_count=$((fps_count + 1))
+                fi
+              done < "$f"
+            done
+            if  $fps_count -gt 0 ; then
+              avg_fps=$(echo "scale=2; $total_fps / $fps_count" | bc 2>/dev/null || echo "N/A")
+              echo " TOTALS: $total_encodes encodes across $(ls summaries/*.log 2>/dev/null | wc -l) test suites"
+              echo " AVERAGE FPS: $avg_fps (across all encodes)"
+            else
+              echo " No encoding summaries found."
+            fi
+          - echo "================================================================"
+
+pipelines:
+  default:
+    - parallel:
+        - step: *build-8bit
+        - step: *build-10bit
+        - step: *build-12bit
+    - step: *build-multilib
+    - parallel:
+        - step: *test-cli-presets
+        - step: *test-rate-control
+        - step: *test-encoding-features
+        - step: *test-analysis-save-load
+        - step: *test-bitstream-validation
+        - step: *test-threading
+        - step: *test-quality-metrics
+        - step: *test-realworld-video
+        - step: *test-10bit-12bit-encoding
+        - step: *test-mcstf-sbrc-interlace
+    - step: *summary-report

x265-4.1.tar/doc/reST/releasenotes.rst -> x265-4.2.tar/doc/reST/releasenotes.rst Changed

@@ -2,6 +2,53 @@
 Release Notes
 *************
 
+Version 4.2
+===========
+
+Release date - 19th April, 2026.
+
+New feature
+-----------
+1. Threaded Motion Estimation (Experimental feature)-It uses a dedicated threadpool to precompute Motion Estimation in parallel.Improves encoding speed upto 1.5x for 1080p & lower resolution on multi core machines with low frequency setting. On high frequency systems or on machines with low number of cores, the overhead of additional Motion estimation work may outweigh parallelism
+
+Enhancements to existing features
+-------------
+
+1.Add new Levels 6.3 to 7.2 specified in ITU-T H.265 (V9) (09/2023) and above
+2.Improve Slices feature with check zeroMv
+3.Enable frame parallelism with MCSTF feature
+4.Updated support to signal AOM FGM params
+5.Improve quality with SBRC feature
+6.Updated DolbyVision P5 VUI defaults
+
+API changes
+-----------
+1. API Support to enable Threaded Motion Estimation(--threaded-me)
+
+Optimizations
+-------------
+1. RISC V optimizations including SAD, SATD, DCT, IDCT, block copy, pixel utilities, SAO, loopfilter, transpose kernels resulting in 2x encoding speed.
+2. ARM SIMD optimizations including the use of NEON and SVE instruction set extensions. The following algorithms now have optimized SIMD implementations: DST, IDCT, SSE, SSD ,intra_pred_planar, pelFilterLumaStrong, interpolation, planecopy, dequant_normal, blockcopy, pixel variance resulting in 8% faster encoding speed compared to v4.1
+
+Bug fixes
+---------
+1. Fix memory leaks (no command line option, SEI buffer, analysis save/load)
+2. Fix chroma qp offset for non yuv444 inputs
+3. Fix max supported input resolution
+4. Fix bugs with ARM SIMD optimizations
+5. Fix Alpha and Multiview feature flag support in x265_config
+6. Fix test harness issues, CMake errors
+7. Fix inconsistent output with aq-motion
+8. Fix crash with hist-scenecut on high bit-depth builds
+9. Fix lookahead concurrency bug
+10. Fix shared link issue (R_X86_64_PC32), yuv recon output issue, rd-refine and dynamic-refine issue, inputs for Windows named pipe,weighted prediction delta_chroma_offset, crf and vbv issue in abr-ladder, psnr and ssim reported with MCSTF feature, internally overflowed VBV variables
+
+Known issues
+------------
+1. Output mismatch between analysis save & load with cutree with reuse level < 10
+2. Inconsistent hash mismatch with abr-ladder feature
+3. Performance regression observed with threaded-me feature on high frequency systems and for higher resolutions (4k)
+
 Version 4.1
 ===========

x265-4.1.tar/source/CMakeLists.txt -> x265-4.2.tar/source/CMakeLists.txt Changed

@@ -6,18 +6,19 @@
         FORCE)
 endif()
 message(STATUS "cmake version ${CMAKE_VERSION}")
-if(POLICY CMP0025)
-    cmake_policy(SET CMP0025 OLD) # report Apple's Clang as just Clang
-endif()
+
 if(POLICY CMP0042)
     cmake_policy(SET CMP0042 NEW) # MACOSX_RPATH
 endif()
-if(POLICY CMP0054)
-    cmake_policy(SET CMP0054 OLD) # Only interpret if() arguments as variables or keywords when unquoted
+
+cmake_minimum_required (VERSION 2.8.8...3.10) # OBJECT libraries require 2.8.8
+
+if (POLICY CMP0075)
+    cmake_policy(SET CMP0075 NEW) # CMAKE_REQUIRED_LIBRARIES warning
 endif()
 
+
 project (x265)
-cmake_minimum_required (VERSION 2.8.8) # OBJECT libraries require 2.8.8
 include(CheckIncludeFiles)
 include(CheckFunctionExists)
 include(CheckSymbolExists)
@@ -31,7 +32,7 @@
 option(STATIC_LINK_CRT "Statically link C and C++ runtimes for release builds" OFF)
 mark_as_advanced(FPROFILE_USE FPROFILE_GENERATE NATIVE_BUILD)
 # X265_BUILD must be incremented each time the public API is changed
-set(X265_BUILD 215)
+set(X265_BUILD 216)
 configure_file("${PROJECT_SOURCE_DIR}/x265.def.in"
                "${PROJECT_BINARY_DIR}/x265.def")
 configure_file("${PROJECT_SOURCE_DIR}/x265_config.h.in"
@@ -48,11 +49,13 @@
 set(X86_ALIASES x86 i386 i686 x86_64 amd64)
 set(ARM_ALIASES armv6l armv7l)
 set(ARM64_ALIASES arm64 arm64e aarch64)
+set(RISCV64_ALIASES riscv64)
 list(FIND X86_ALIASES "${SYSPROC}" X86MATCH)
 list(FIND ARM_ALIASES "${SYSPROC}" ARMMATCH)
 list(FIND ARM64_ALIASES "${SYSPROC}" ARM64MATCH)
 set(POWER_ALIASES powerpc64 powerpc64le ppc64 ppc64le)
 list(FIND POWER_ALIASES "${SYSPROC}" POWERMATCH)
+list(FIND RISCV64_ALIASES "${SYSPROC}" RISCV64MATCH)
 if(X86MATCH GREATER "-1")
     set(X86 1)
     add_definitions(-DX265_ARCH_X86=1)
@@ -99,6 +102,7 @@
     option(ENABLE_NEON_I8MM "Enable Neon I8MM" ON)
     option(ENABLE_SVE "Enable SVE" ON)
     option(ENABLE_SVE2 "Enable SVE2" ON)
+    option(ENABLE_SVE2_BITPERM "Enable SVE2 BitPerm" ON)
 
     # Compiler flags for AArch64 extensions.
     set(AARCH64_NEON_FLAG "-march=armv8-a")
@@ -109,11 +113,34 @@
     set(AARCH64_SVE_FLAG "-march=armv8.2-a+dotprod+i8mm+sve")
     # SVE2 is only available from Armv9.0, and armv9-a implies +dotprod and +sve.
     set(AARCH64_SVE2_FLAG "-march=armv9-a+i8mm+sve2")
+    # SVE2 BitPerm implies +dotprod, +sve, and +sve2.
+    set(AARCH64_SVE2_BITPERM_FLAG "-march=armv9-a+i8mm+sve2-bitperm")
+elseif(RISCV64MATCH GREATER "-1")
+    message(STATUS "Detected RISCV64 target processor")
+    set(RISCV64 1)
+
+    option(RISCV64_RUNTIME_CPU_DETECT "Enable RISCV64 run-time CPU feature detection" ON)
+
+    option(ENABLE_RVV "Enable RVV" ON)
+
+    # Compiler flags only for riscv64 intrinsic file.
+    set(RISCV64_INTRINSIC_FLAG "-march=rv64gcv")
 else()
     message(STATUS "CMAKE_SYSTEM_PROCESSOR value `${CMAKE_SYSTEM_PROCESSOR}` is unknown")
     message(STATUS "Please add this value near ${CMAKE_CURRENT_LIST_FILE}:${CMAKE_CURRENT_LIST_LINE}")
 endif()
 
+if(X64)
+    option(ENABLE_CET "Enable Control-flow Enforcement" OFF)
+    if(ENABLE_CET)
+        add_definitions(-DENABLE_CET=1)
+        list(APPEND ASM_FLAGS "-DENABLE_CET=1")
+    else()
+        add_definitions(-DENABLE_CET=0)
+        list(APPEND ASM_FLAGS "-DENABLE_CET=0")
+    endif()
+endif(X64)
+
 if(UNIX)
     list(APPEND PLATFORM_LIBS pthread)
     find_library(LIBRT rt)
@@ -152,9 +179,16 @@
     if(ENABLE_LIBVMAF)
         add_definitions(-DENABLE_LIBVMAF)
     endif()
+    if(X64)
+        option(ENABLE_CET "Enable Control-flow Enforcement" OFF)
+        if(ENABLE_CET)
+            add_definitions(-DENABLE_CET)
+            list(APPEND ASM_FLAGS "-DENABLE_CET=1")
+        endif()
+    endif(X64)
 endif(UNIX)
 
-if((X64 AND NOT WIN32) OR ARM64)
+if((X64 AND NOT WIN32) OR ARM64 OR PPC64 OR RISCV64)
     option(ENABLE_PIC "Enable Position Independent Code" ON)
 else()
     option(ENABLE_PIC "Enable Position Independent Code" OFF)
@@ -168,7 +202,7 @@
   add_definitions(-DMACOS=1)
 endif()
 
-if(${CMAKE_CXX_COMPILER_ID} STREQUAL "Clang")
+if(${CMAKE_CXX_COMPILER_ID} STREQUAL "Clang" OR ${CMAKE_CXX_COMPILER_ID} STREQUAL "AppleClang")
     set(CLANG 1)
 endif()
 if(${CMAKE_CXX_COMPILER_ID} STREQUAL "Intel")
@@ -295,12 +329,14 @@
             set(CPU_HAS_NEON_I8MM 1)
             set(CPU_HAS_SVE 1)
             set(CPU_HAS_SVE2 1)
+            set(CPU_HAS_SVE2_BITPERM 1)
         else()
             if(CMAKE_SYSTEM_NAME MATCHES "Linux|Darwin")
                 find_package(NEON_DOTPROD)
                 find_package(NEON_I8MM)
                 find_package(SVE)
                 find_package(SVE2)
+                find_package(SVE2_BITPERM)
             else()
                 message(STATUS "Compile-time CPU feature detection unsupported on this platform")
             endif()
@@ -311,6 +347,11 @@
             string(APPEND CMAKE_REQUIRED_FLAGS " ${AARCH64_SVE_FLAG}")
             set(OLD_CMAKE_TRY_COMPILE_TARGET_TYPE ${CMAKE_TRY_COMPILE_TARGET_TYPE})
             set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
+            # When compilation target is a STATIC_LIBRARY, the LINK_OPTIONS are
+            # passed to the archiver, so we must backup, clear and restore these.
+            # https://gitlab.kitware.com/cmake/cmake/-/issues/23454
+            set(OLD_CMAKE_REQUIRED_LINK_OPTIONS ${CMAKE_REQUIRED_LINK_OPTIONS})
+            set(CMAKE_REQUIRED_LINK_OPTIONS "")
 
             # Check whether the compiler can compile SVE functions that require
             # backup/restore of SVE registers according to AAPCS.
@@ -340,6 +381,7 @@
 
             set(CMAKE_REQUIRED_FLAGS ${OLD_CMAKE_REQUIRED_FLAGS})
             set(CMAKE_TRY_COMPILE_TARGET_TYPE ${OLD_CMAKE_TRY_COMPILE_TARGET_TYPE})
+            set(CMAKE_REQUIRED_LINK_OPTIONS ${OLD_CMAKE_REQUIRED_LINK_OPTIONS})
             if (SVE_COMPILATION_C_TEST_COMPILED AND SVE_COMPILATION_CXX_TEST_COMPILED)
                 if (SVE_HEADER_C_TEST_COMPILED AND SVE_HEADER_CXX_TEST_COMPILED)
                     add_definitions(-DHAVE_SVE_BRIDGE=1)
@@ -376,6 +418,11 @@
         if(NOT ENABLE_SVE2)
             message(STATUS "Disabling SVE2")
             set(CPU_HAS_SVE2 0)
+            set(ENABLE_SVE2_BITPERM 0)
+        endif()
+        if(NOT ENABLE_SVE2_BITPERM)
+            message(STATUS "Disabling SVE2 BitPerm")
+            set(CPU_HAS_SVE2_BITPERM 0)
         endif()
 
         if(CPU_HAS_NEON)
@@ -398,6 +445,10 @@
             message(STATUS "Found SVE2")
             add_definitions(-DHAVE_SVE2=1)
         endif()
+        if(CPU_HAS_SVE2_BITPERM)
+            message(STATUS "Found SVE2 BitPerm")
+            add_definitions(-DHAVE_SVE2_BITPERM=1)
+        endif()
         set(ARM_ARGS -O3)
         # Do not allow implicit vector type conversions in Clang builds (this
         # is already the default in GCC builds).
@@ -406,6 +457,87 @@
             set(ARM_ARGS ${ARM_ARGS} -flax-vector-conversions=none)
         endif()
     endif()
+    if(RISCV64)
+        add_definitions(-DX265_ARCH_RISCV64=1)
+
+        if (RISCV64_RUNTIME_CPU_DETECT)
+            add_definitions(-DRISCV64_RUNTIME_CPU_DETECT=1)
+            message(STATUS "Configuring build for run-time CPU feature detection")
+        endif()
+
+        if(RISCV64_RUNTIME_CPU_DETECT OR CROSS_COMPILE_RISCV64)
+            # Add all extensions when compiling for run-time CPU feature detection or cross compiling.
+            set(CPU_HAS_RVV 1)
+        else()
+            if(CMAKE_SYSTEM_NAME MATCHES "Linux|Darwin")
+                find_package(RVV)
+            else()
+                message(STATUS "Compile-time CPU feature detection unsupported on this platform")
+            endif()
+        endif()
+
+        if(ENABLE_RVV)
+            set(OLD_CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS})
+            set(OLD_CMAKE_TRY_COMPILE_TARGET_TYPE ${CMAKE_TRY_COMPILE_TARGET_TYPE})
+            set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
+            # When compilation target is a STATIC_LIBRARY, the LINK_OPTIONS are
+            # passed to the archiver, so we must backup, clear and restore these.
+            # https://gitlab.kitware.com/cmake/cmake/-/issues/23454
+            set(OLD_CMAKE_REQUIRED_LINK_OPTIONS ${CMAKE_REQUIRED_LINK_OPTIONS})
+            set(CMAKE_REQUIRED_LINK_OPTIONS "")
+
+            # Check whether the compiler can compile RVV functions that require
+            # backup/restore of RVV registers according to AAPCS.
+            # https://github.com/llvm/llvm-project/issues/80009.
+            set(RVV_COMPILATION_TEST 
+int main() {
+  asm volatile(".option arch, +v; vsetvli t0, a0, e32, m1, ta, ma");
+  return 0;
+})
+
+            check_c_source_compiles("${RVV_COMPILATION_TEST}" RVV_COMPILATION_C_TEST_COMPILED)
+            check_cxx_source_compiles("${RVV_COMPILATION_TEST}" RVV_COMPILATION_CXX_TEST_COMPILED)
+
+            set(CMAKE_REQUIRED_FLAGS ${OLD_CMAKE_REQUIRED_FLAGS})
+            set(CMAKE_TRY_COMPILE_TARGET_TYPE ${OLD_CMAKE_TRY_COMPILE_TARGET_TYPE})
+            set(CMAKE_REQUIRED_LINK_OPTIONS ${OLD_CMAKE_REQUIRED_LINK_OPTIONS})
+            if (NOT(RVV_COMPILATION_C_TEST_COMPILED AND RVV_COMPILATION_CXX_TEST_COMPILED))
+                set(ENABLE_RVV OFF CACHE BOOL "" FORCE)
+                message(STATUS "Disabling RVV")
+            endif()
+        endif()
+
+        if(NOT ENABLE_RVV)
+            message(STATUS "Disabling RVV")
+            set(CPU_HAS_RVV 0)
+        endif()
+
+        set(CPU_HAS_RVV_INTRINSIC 0)
+        if(CPU_HAS_RVV)
+            message(STATUS "Found RVV")
+            add_definitions(-DHAVE_RVV=1)
+
+            set(RVV_INTRINSIC_TEST 
+#include <riscv_vector.h>
+#include <stdint.h>
+int main() {
+    size_t vl = __riscv_vsetvl_e32m1(10);
+    return 0;
+})
+            set(CMAKE_REQUIRED_FLAGS ${RISCV64_INTRINSIC_FLAG})
+            check_c_source_compiles("${RVV_INTRINSIC_TEST}" RVV_INTRINSIC_C_TEST_COMPILED)
+            check_cxx_source_compiles("${RVV_INTRINSIC_TEST}" RVV_INTRINSIC_CXX_TEST_COMPILED)
+            if (RVV_INTRINSIC_C_TEST_COMPILED AND RVV_INTRINSIC_CXX_TEST_COMPILED)
+                set(CPU_HAS_RVV_INTRINSIC 1)
+                add_definitions(-DHAVE_RVV_INTRINSIC=1)
+                message(STATUS "Found RVV intrinsic")
+            else()
+                message(STATUS "Disabling RVV intrinsic")
+            endif()
+        endif()
+
+        set(RISCV64_ARGS -O2)
+    endif()
     if(ENABLE_PIC)
         list(APPEND ARM_ARGS -DPIC -fPIC)
     endif()
@@ -458,7 +590,7 @@
     if (CC_HAS_FAST_MATH)
         add_definitions(-ffast-math)
     endif()
-    if (NOT (ARM64 OR CROSS_COMPILE_ARM64))
+    if (NOT (ARM64 OR CROSS_COMPILE_ARM64 OR RISCV64 OR CROSS_COMPILE_RISCV64))
         check_cxx_compiler_flag(-mstackrealign CC_HAS_STACK_REALIGN)
         if (CC_HAS_STACK_REALIGN)
             add_definitions(-mstackrealign)
@@ -477,6 +609,12 @@
         # linked.
         list(APPEND LINKER_OPTIONS "-fsanitize=${FSANITIZE}")
     endif()
+    if(ENABLE_CET)
+        check_cxx_compiler_flag(-fcf-protection=full CC_HAS_CET_FLAG)
+        if(CC_HAS_CET_FLAG)
+            add_definitions(-fcf-protection=full)
+        endif()
+    endif()
     option(ENABLE_AGGRESSIVE_CHECKS "Enable stack protection and -ftrapv" OFF)
     if(ENABLE_AGGRESSIVE_CHECKS)
         # use with care, -ftrapv can cause testbench SIGILL exceptions
@@ -508,6 +646,8 @@
     if (NOT ENABLE_NEON)
         set(ENABLE_ASSEMBLY OFF CACHE BOOL "" FORCE)
     endif()
+elseif((RISCV64 OR CROSS_COMPILE_RISCV64) AND ENABLE_RVV)
+    option(ENABLE_ASSEMBLY "Enable use of assembly coded primitives" ON)
 elseif(NASM_FOUND AND X86)
     if (NASM_VERSION_STRING VERSION_LESS "2.13.0")
         message(STATUS "Nasm version ${NASM_VERSION_STRING} is too old. 2.13.0 or later required")
@@ -544,7 +684,7 @@
 endif(EXTRA_LIB)
 mark_as_advanced(EXTRA_LIB EXTRA_LINK_FLAGS)
 
-if(X64 OR ARM64 OR PPC64)
+if(X64 OR ARM64 OR PPC64 OR RISCV64)
     # NOTE: We only officially support high-bit-depth compiles of x265
     # on 64bit architectures. Main10 plus large resolution plus slow
     # preset plus 32bit address space usually means malloc failure.  You
@@ -553,7 +693,7 @@
     # license" so to speak.  If it breaks you get to keep both halves.
     # You will need to disable assembly manually.
     option(HIGH_BIT_DEPTH "Store pixel samples as 16bit values (Main10/Main12)" OFF)
-endif(X64 OR ARM64 OR PPC64)
+endif(X64 OR ARM64 OR PPC64 OR RISCV64)
 if(HIGH_BIT_DEPTH)
     option(MAIN12 "Support Main12 instead of Main10" OFF)
     if(MAIN12)
@@ -600,7 +740,7 @@
 endif()
 add_definitions(-DX265_NS=${X265_NS})
 
-if(ARM64)
+if(ARM64 OR RISCV64)
   if(HIGH_BIT_DEPTH)
     if(MAIN12)
       list(APPEND ASM_FLAGS -DHIGH_BIT_DEPTH=1 -DBIT_DEPTH=12 -DX265_NS=${X265_NS})
@@ -610,13 +750,12 @@
   else()
     list(APPEND ASM_FLAGS -DHIGH_BIT_DEPTH=0 -DBIT_DEPTH=8 -DX265_NS=${X265_NS})
   endif()
-endif(ARM64)
+endif(ARM64 OR RISCV64)
 
 option(WARNINGS_AS_ERRORS "Stop compiles on first warning" OFF)
 if(WARNINGS_AS_ERRORS)
     if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
         add_definitions(-Werror)
-        add_definitions(-Wno-unused)
     elseif(MSVC)
         add_definitions(/WX)
     endif()
@@ -696,17 +835,28 @@
 option(ENABLE_ALPHA "Enable alpha encoding in x265" OFF)
 if(ENABLE_ALPHA)
     add_definitions(-DENABLE_ALPHA)
+    set(ENABLE_ALPHA 1)
+else()
+    set(ENABLE_ALPHA 0)
 endif()
 
 option(ENABLE_MULTIVIEW "Enable Multi-view encoding in HEVC" OFF)
 if(ENABLE_MULTIVIEW)
     add_definitions(-DENABLE_MULTIVIEW)
+    set(ENABLE_MULTIVIEW 1)
+else()
+    set(ENABLE_MULTIVIEW 0)
 endif()
 
 option(ENABLE_SCC_EXT "Enable screen content coding extension in HEVC" OFF)
 if(ENABLE_SCC_EXT)
     add_definitions(-DENABLE_SCC_EXT)
+    set(ENABLE_SCC_EXT 1)
+else()
+    set(ENABLE_SCC_EXT 0)
 endif()
+configure_file("${PROJECT_SOURCE_DIR}/x265_config.h.in"
+               "${PROJECT_BINARY_DIR}/x265_config.h")
 
 add_subdirectory(encoder)
 add_subdirectory(common)
@@ -723,6 +873,9 @@
     if(ARM OR CROSS_COMPILE_ARM)
     # compile ARM arch asm files here
         enable_language(ASM)
+        if(APPLE)
+            set(ARM_ARGS ${ARM_ARGS} -arch ${CMAKE_OSX_ARCHITECTURES})
+        endif()
         foreach(ASM ${ARM_ASMS})
 			set(ASM_SRC ${CMAKE_CURRENT_SOURCE_DIR}/common/arm/${ASM})
             list(APPEND ASM_SRCS ${ASM_SRC})
@@ -782,6 +935,31 @@
                     DEPENDS ${ASM_SRC})
             endforeach()
         endif()
+        if(CPU_HAS_SVE2_BITPERM)
+            foreach(ASM ${ARM_ASMS_SVE2_BITPERM})
+                set(ASM_SRC ${CMAKE_CURRENT_SOURCE_DIR}/common/aarch64/${ASM})
+                list(APPEND ASM_SRCS ${ASM_SRC})
+                list(APPEND ASM_OBJS ${ASM}.${SUFFIX})
+                add_custom_command(
+                    OUTPUT ${ASM}.${SUFFIX}
+                    COMMAND ${CMAKE_CXX_COMPILER}
+                    ARGS ${ARM_ARGS} ${ASM_FLAGS} ${AARCH64_SVE2_BITPERM_FLAG} -c ${ASM_SRC} -o ${ASM}.${SUFFIX}
+                    DEPENDS ${ASM_SRC})
+            endforeach()
+        endif()
+    elseif(RISCV64 OR CROSS_COMPILE_RISCV64)
+    # compile RISCV64 arch asm files here
+        enable_language(ASM)
+        foreach(ASM ${RISCV64_ASMS})
+            set(ASM_SRC ${CMAKE_CURRENT_SOURCE_DIR}/common/riscv64/${ASM})
+            list(APPEND ASM_SRCS ${ASM_SRC})
+            list(APPEND ASM_OBJS ${ASM}.${SUFFIX})
+            add_custom_command(
+                OUTPUT ${ASM}.${SUFFIX}
+                COMMAND ${CMAKE_CXX_COMPILER}
+                ARGS ${RISCV64_ARGS} ${ASM_FLAGS} -c ${ASM_SRC} -o ${ASM}.${SUFFIX}
+                DEPENDS ${ASM_SRC})
+        endforeach()
     elseif(X86)
     # compile X86 arch asm files here
         foreach(ASM ${MSVC_ASMS})

x265-4.1.tar/source/abrEncApp.cpp -> x265-4.2.tar/source/abrEncApp.cpp Changed

x265-4.1.tar/source/cmake/FindNasm.cmake -> x265-4.2.tar/source/cmake/FindNasm.cmake Changed

x265-4.1.tar/source/cmake/FindNuma.cmake -> x265-4.2.tar/source/cmake/FindNuma.cmake Changed

x265-4.2.tar/source/cmake/FindSVE2_BITPERM.cmake Added

x265-4.1.tar/source/common/CMakeLists.txt -> x265-4.2.tar/source/common/CMakeLists.txt Changed

@@ -105,23 +105,25 @@
 
     # Add Arm intrinsics files here.
     set(C_SRCS_NEON asm-primitives.cpp pixel-prim.h pixel-prim.cpp filter-prim.h filter-prim.cpp dct-prim.h dct-prim.cpp loopfilter-prim.cpp loopfilter-prim.h intrapred-prim.cpp arm64-utils.cpp arm64-utils.h fun-decls.h sao-prim.cpp  mem-neon.h)
-    set(C_SRCS_NEON_DOTPROD filter-neon-dotprod.cpp)
+    set(C_SRCS_NEON_DOTPROD filter-neon-dotprod.cpp pixel-prim-neon-dotprod.cpp)
     set(C_SRCS_NEON_I8MM filter-neon-i8mm.cpp)
-    set(C_SRCS_SVE sao-prim-sve.cpp dct-prim-sve.cpp)
+    set(C_SRCS_SVE sao-prim-sve.cpp dct-prim-sve.cpp filter-prim-sve.h filter-prim-sve.cpp pixel-prim-sve.cpp)
     set(C_SRCS_SVE2 sao-prim-sve2.cpp)
     enable_language(ASM)
 
     # Add Arm assembly files here.
-    set(A_SRCS asm.S mc-a.S mc-a-common.S sad-a.S pixel-util.S pixel-util-common.S p2s.S p2s-common.S blockcopy8.S blockcopy8-common.S ssd-a.S ssd-a-common.S intrapred.S dct.S)
+    set(A_SRCS asm.S mc-a.S mc-a-common.S sad-a.S pixel-util.S pixel-util-common.S p2s.S p2s-common.S blockcopy8.S ssd-a.S ssd-a-common.S intrapred.S dct.S)
     set(A_SRCS_NEON_DOTPROD sad-neon-dotprod.S ssd-neon-dotprod.S)
-    set(A_SRCS_SVE asm-sve.S blockcopy8-sve.S p2s-sve.S pixel-util-sve.S)
-    set(A_SRCS_SVE2 mc-a-sve2.S pixel-util-sve2.S ssd-a-sve2.S)
+    set(A_SRCS_SVE asm-sve.S p2s-sve.S pixel-util-sve.S ssd-a-sve.S)
+    set(A_SRCS_SVE2 mc-a-sve2.S pixel-util-sve2.S)
+    set(A_SRCS_SVE2_BITPERM pixel-util-sve2-bitperm.S)
     set(VEC_PRIMITIVES)
 
-    set(ARM_ASMS "${A_SRCS}" CACHE INTERNAL "ARM Assembly Sources")
-    set(ARM_ASMS_SVE "${A_SRCS_SVE}" CACHE INTERNAL "ARM Assembly Sources that use SVE instruction set")
-    set(ARM_ASMS_SVE2 "${A_SRCS_SVE2}" CACHE INTERNAL "ARM Assembly Sources that use SVE2 instruction set")
+    set(ARM_ASMS "${A_SRCS}" CACHE INTERNAL "Arm Assembly Sources")
     set(ARM_ASMS_NEON_DOTPROD "${A_SRCS_NEON_DOTPROD}" CACHE INTERNAL "Arm Assembly Sources that use the Neon DotProd extension")
+    set(ARM_ASMS_SVE "${A_SRCS_SVE}" CACHE INTERNAL "Arm Assembly Sources that use the SVE extension")
+    set(ARM_ASMS_SVE2 "${A_SRCS_SVE2}" CACHE INTERNAL "Arm Assembly Sources that use the SVE2 extension")
+    set(ARM_ASMS_SVE2_BITPERM "${A_SRCS_SVE2_BITPERM}" CACHE INTERNAL "Arm Assembly Sources that use the SVE2 BitPerm extension")
     foreach(SRC ${C_SRCS_NEON})
         set(ASM_PRIMITIVES ${ASM_PRIMITIVES} aarch64/${SRC})
         set_source_files_properties(aarch64/${SRC} PROPERTIES COMPILE_FLAGS ${AARCH64_NEON_FLAG})
@@ -162,6 +164,39 @@
     endif()
 endif(ENABLE_ASSEMBLY AND (ARM64 OR CROSS_COMPILE_ARM64))
 
+if(ENABLE_ASSEMBLY AND (RISCV64 OR CROSS_COMPILE_RISCV64))
+    if(GCC AND (CMAKE_CXX_FLAGS_RELEASE MATCHES "-O3"))
+        message(STATUS "Detected CXX compiler using -O3 optimization level")
+        add_definitions(-DAUTO_VECTORIZE=1)
+    endif()
+
+    set(ASM_PRIMITIVES ${ASM_PRIMITIVES} riscv64/asm-primitives.cpp)
+
+    # Add riscv64 intrinsics files here.
+    set(INTRINSIC_SRCS_RVV pixel-prim.cpp sao-prim.cpp filter-prim.cpp intrapred-prim.cpp riscv64_utils.cpp)
+    if(CPU_HAS_RVV_INTRINSIC)
+        foreach(SRC ${INTRINSIC_SRCS_RVV})
+            set(ASM_PRIMITIVES ${ASM_PRIMITIVES} riscv64/${SRC})
+            set_source_files_properties(riscv64/${SRC} PROPERTIES COMPILE_FLAGS ${RISCV64_INTRINSIC_FLAG})
+        endforeach()
+    endif()
+
+    enable_language(ASM)
+    source_group(Assembly FILES ${ASM_PRIMITIVES})
+
+    # Add riscv64 assembly files here.
+    set(A_SRCS asm.S blockcopy8.S dct.S sad-a.S ssd-a.S pixel-util.S mc-a.S p2s.S sao.S loopfilter.S intrapred.S riscv64_utils.S)
+    set(VEC_PRIMITIVES)
+
+    if(CPU_HAS_RVV)
+        set(RISCV64_ASMS "${A_SRCS}" CACHE INTERNAL "RISCV64 Assembly Sources")
+    endif()
+
+    if(RISCV64_WARNINGS_AS_ERRORS)
+        set_source_files_properties(${ASM_PRIMITIVES} PROPERTIES COMPILE_FLAGS -Werror)
+    endif()
+endif(ENABLE_ASSEMBLY AND (RISCV64 OR CROSS_COMPILE_RISCV64))
+
 if(POWER)
     set_source_files_properties(version.cpp PROPERTIES COMPILE_FLAGS -DX265_VERSION=${X265_VERSION})
     if(ENABLE_ALTIVEC)

x265-4.1.tar/source/common/aarch64/asm-primitives.cpp -> x265-4.2.tar/source/common/aarch64/asm-primitives.cpp Changed

@@ -380,6 +380,7 @@
 
 #include "pixel-prim.h"
 #include "filter-prim.h"
+#include "filter-prim-sve.h"
 #include "dct-prim.h"
 #include "loopfilter-prim.h"
 #include "intrapred-prim.h"
@@ -403,82 +404,6 @@
     ALL_CHROMA_444_PU(p2sNONALIGNED, filterPixelToShort, neon);
     ALL_LUMA_PU(convert_p2sNONALIGNED, filterPixelToShort, neon);
 
-#if !HIGH_BIT_DEPTH
-    // Blockcopy_pp
-    ALL_LUMA_PU(copy_pp, blockcopy_pp, neon);
-    ALL_CHROMA_420_PU(copy_pp, blockcopy_pp, neon);
-    ALL_CHROMA_422_PU(copy_pp, blockcopy_pp, neon);
-    p.cuBLOCK_4x4.copy_pp   = PFX(blockcopy_pp_4x4_neon);
-    p.cuBLOCK_8x8.copy_pp   = PFX(blockcopy_pp_8x8_neon);
-    p.cuBLOCK_16x16.copy_pp = PFX(blockcopy_pp_16x16_neon);
-    p.cuBLOCK_32x32.copy_pp = PFX(blockcopy_pp_32x32_neon);
-    p.cuBLOCK_64x64.copy_pp = PFX(blockcopy_pp_64x64_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.copy_pp = PFX(blockcopy_pp_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.copy_pp = PFX(blockcopy_pp_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_pp = PFX(blockcopy_pp_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_pp = PFX(blockcopy_pp_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_pp = PFX(blockcopy_pp_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_pp = PFX(blockcopy_pp_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_pp = PFX(blockcopy_pp_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_pp = PFX(blockcopy_pp_32x64_neon);
-
-#endif // !HIGH_BIT_DEPTH
-
-    // Blockcopy_ss
-    p.cuBLOCK_4x4.copy_ss   = PFX(blockcopy_ss_4x4_neon);
-    p.cuBLOCK_8x8.copy_ss   = PFX(blockcopy_ss_8x8_neon);
-    p.cuBLOCK_16x16.copy_ss = PFX(blockcopy_ss_16x16_neon);
-    p.cuBLOCK_32x32.copy_ss = PFX(blockcopy_ss_32x32_neon);
-    p.cuBLOCK_64x64.copy_ss = PFX(blockcopy_ss_64x64_neon);
-
-    // Blockcopy_ps
-    p.cuBLOCK_4x4.copy_ps   = PFX(blockcopy_ps_4x4_neon);
-    p.cuBLOCK_8x8.copy_ps   = PFX(blockcopy_ps_8x8_neon);
-    p.cuBLOCK_16x16.copy_ps = PFX(blockcopy_ps_16x16_neon);
-    p.cuBLOCK_32x32.copy_ps = PFX(blockcopy_ps_32x32_neon);
-    p.cuBLOCK_64x64.copy_ps = PFX(blockcopy_ps_64x64_neon);
-
-    // Blockcopy_sp
-    p.cuBLOCK_4x4.copy_sp   = PFX(blockcopy_sp_4x4_neon);
-    p.cuBLOCK_8x8.copy_sp   = PFX(blockcopy_sp_8x8_neon);
-    p.cuBLOCK_16x16.copy_sp = PFX(blockcopy_sp_16x16_neon);
-    p.cuBLOCK_32x32.copy_sp = PFX(blockcopy_sp_32x32_neon);
-    p.cuBLOCK_64x64.copy_sp = PFX(blockcopy_sp_64x64_neon);
-
-    // chroma blockcopy_ss
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.copy_ss   = PFX(blockcopy_ss_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.copy_ss   = PFX(blockcopy_ss_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_ss = PFX(blockcopy_ss_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_ss = PFX(blockcopy_ss_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_ss   = PFX(blockcopy_ss_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_ss  = PFX(blockcopy_ss_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_ss = PFX(blockcopy_ss_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_ss = PFX(blockcopy_ss_32x64_neon);
-
-    // chroma blockcopy_ps
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.copy_ps   = PFX(blockcopy_ps_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.copy_ps   = PFX(blockcopy_ps_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_ps = PFX(blockcopy_ps_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_ps = PFX(blockcopy_ps_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_ps   = PFX(blockcopy_ps_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_ps  = PFX(blockcopy_ps_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_ps = PFX(blockcopy_ps_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_ps = PFX(blockcopy_ps_32x64_neon);
-
-    // chroma blockcopy_sp
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.copy_sp   = PFX(blockcopy_sp_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.copy_sp   = PFX(blockcopy_sp_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_sp = PFX(blockcopy_sp_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_sp = PFX(blockcopy_sp_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_sp   = PFX(blockcopy_sp_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_sp  = PFX(blockcopy_sp_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_sp = PFX(blockcopy_sp_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_sp = PFX(blockcopy_sp_32x64_neon);
-
-    // Block_fill
-    ALL_LUMA_TU(blockfill_sALIGNED, blockfill_s, neon);
-    ALL_LUMA_TU(blockfill_sNONALIGNED, blockfill_s, neon);
-
     // copy_count
     p.cuBLOCK_4x4.copy_cnt     = PFX(copy_cnt_4_neon);
     p.cuBLOCK_8x8.copy_cnt     = PFX(copy_cnt_8_neon);
@@ -491,52 +416,6 @@
     p.cuBLOCK_16x16.count_nonzero   = PFX(count_nonzero_16_neon);
     p.cuBLOCK_32x32.count_nonzero   = PFX(count_nonzero_32_neon);
 
-    // cpy2Dto1D_shl
-    p.cuBLOCK_4x4.cpy2Dto1D_shl   = PFX(cpy2Dto1D_shl_4x4_neon);
-    p.cuBLOCK_8x8.cpy2Dto1D_shl   = PFX(cpy2Dto1D_shl_8x8_neon);
-    p.cuBLOCK_16x16.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_16x16_neon);
-    p.cuBLOCK_32x32.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_32x32_neon);
-    p.cuBLOCK_64x64.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_64x64_neon);
-
-    // cpy2Dto1D_shr
-    p.cuBLOCK_4x4.cpy2Dto1D_shr   = PFX(cpy2Dto1D_shr_4x4_neon);
-    p.cuBLOCK_8x8.cpy2Dto1D_shr   = PFX(cpy2Dto1D_shr_8x8_neon);
-    p.cuBLOCK_16x16.cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_16x16_neon);
-    p.cuBLOCK_32x32.cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_32x32_neon);
-
-    // cpy1Dto2D_shl
-    p.cuBLOCK_4x4.cpy1Dto2D_shlALIGNED      = PFX(cpy1Dto2D_shl_4x4_neon);
-    p.cuBLOCK_8x8.cpy1Dto2D_shlALIGNED      = PFX(cpy1Dto2D_shl_8x8_neon);
-    p.cuBLOCK_16x16.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_16x16_neon);
-    p.cuBLOCK_32x32.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_32x32_neon);
-    p.cuBLOCK_64x64.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_64x64_neon);
-
-    p.cuBLOCK_4x4.cpy1Dto2D_shlNONALIGNED   = PFX(cpy1Dto2D_shl_4x4_neon);
-    p.cuBLOCK_8x8.cpy1Dto2D_shlNONALIGNED   = PFX(cpy1Dto2D_shl_8x8_neon);
-    p.cuBLOCK_16x16.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_16x16_neon);
-    p.cuBLOCK_32x32.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_32x32_neon);
-    p.cuBLOCK_64x64.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_64x64_neon);
-
-    // cpy1Dto2D_shr
-    p.cuBLOCK_4x4.cpy1Dto2D_shr   = PFX(cpy1Dto2D_shr_4x4_neon);
-    p.cuBLOCK_8x8.cpy1Dto2D_shr   = PFX(cpy1Dto2D_shr_8x8_neon);
-    p.cuBLOCK_16x16.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_16x16_neon);
-    p.cuBLOCK_32x32.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_32x32_neon);
-    p.cuBLOCK_64x64.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_64x64_neon);
-
-#if !HIGH_BIT_DEPTH
-    // pixel_avg_pp
-    ALL_LUMA_PU(pixelavg_ppNONALIGNED, pixel_avg_pp, neon);
-    ALL_LUMA_PU(pixelavg_ppALIGNED, pixel_avg_pp, neon);
-
-    // addAvg
-    ALL_LUMA_PU(addAvgNONALIGNED, addAvg, neon);
-    ALL_LUMA_PU(addAvgALIGNED, addAvg, neon);
-    ALL_CHROMA_420_PU(addAvgNONALIGNED, addAvg, neon);
-    ALL_CHROMA_422_PU(addAvgNONALIGNED, addAvg, neon);
-    ALL_CHROMA_420_PU(addAvgALIGNED, addAvg, neon);
-    ALL_CHROMA_422_PU(addAvgALIGNED, addAvg, neon);
-
     // sad
     ALL_LUMA_PU(sad, pixel_sad, neon);
     ALL_LUMA_PU(sad_x3, sad_x3, neon);
@@ -570,17 +449,18 @@
     p.cuBLOCK_8x8.ssd_sNONALIGNED   = PFX(pixel_ssd_s_8x8_neon);
     p.cuBLOCK_16x16.ssd_sNONALIGNED = PFX(pixel_ssd_s_16x16_neon);
     p.cuBLOCK_32x32.ssd_sNONALIGNED = PFX(pixel_ssd_s_32x32_neon);
+    p.cuBLOCK_64x64.ssd_sNONALIGNED = PFX(pixel_ssd_s_64x64_neon);
 
     p.cuBLOCK_4x4.ssd_sALIGNED   = PFX(pixel_ssd_s_4x4_neon);
     p.cuBLOCK_8x8.ssd_sALIGNED   = PFX(pixel_ssd_s_8x8_neon);
     p.cuBLOCK_16x16.ssd_sALIGNED = PFX(pixel_ssd_s_16x16_neon);
     p.cuBLOCK_32x32.ssd_sALIGNED = PFX(pixel_ssd_s_32x32_neon);
+    p.cuBLOCK_64x64.ssd_sALIGNED = PFX(pixel_ssd_s_64x64_neon);
 
-    // pixel_var
-    p.cuBLOCK_8x8.var   = PFX(pixel_var_8x8_neon);
-    p.cuBLOCK_16x16.var = PFX(pixel_var_16x16_neon);
-    p.cuBLOCK_32x32.var = PFX(pixel_var_32x32_neon);
-    p.cuBLOCK_64x64.var = PFX(pixel_var_64x64_neon);
+#if !HIGH_BIT_DEPTH
+    // pixel_avg_pp
+    ALL_LUMA_PU(pixelavg_ppNONALIGNED, pixel_avg_pp, neon);
+    ALL_LUMA_PU(pixelavg_ppALIGNED, pixel_avg_pp, neon);
 
     // calc_Residual
     p.cuBLOCK_4x4.calcresidualNONALIGNED   = PFX(getResidual4_neon);
@@ -610,38 +490,6 @@
     p.chromaX265_CSP_I422.cuBLOCK_422_16x32.sub_ps = PFX(pixel_sub_ps_16x32_neon);
     p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sub_ps = PFX(pixel_sub_ps_32x64_neon);
 
-    // pixel_add_ps
-    p.cuBLOCK_4x4.add_psNONALIGNED   = PFX(pixel_add_ps_4x4_neon);
-    p.cuBLOCK_8x8.add_psNONALIGNED   = PFX(pixel_add_ps_8x8_neon);
-    p.cuBLOCK_16x16.add_psNONALIGNED = PFX(pixel_add_ps_16x16_neon);
-    p.cuBLOCK_32x32.add_psNONALIGNED = PFX(pixel_add_ps_32x32_neon);
-    p.cuBLOCK_64x64.add_psNONALIGNED = PFX(pixel_add_ps_64x64_neon);
-
-    p.cuBLOCK_4x4.add_psALIGNED   = PFX(pixel_add_ps_4x4_neon);
-    p.cuBLOCK_8x8.add_psALIGNED   = PFX(pixel_add_ps_8x8_neon);
-    p.cuBLOCK_16x16.add_psALIGNED = PFX(pixel_add_ps_16x16_neon);
-    p.cuBLOCK_32x32.add_psALIGNED = PFX(pixel_add_ps_32x32_neon);
-    p.cuBLOCK_64x64.add_psALIGNED = PFX(pixel_add_ps_64x64_neon);
-
-    // chroma add_ps
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.add_psNONALIGNED   = PFX(pixel_add_ps_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.add_psNONALIGNED   = PFX(pixel_add_ps_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.add_psNONALIGNED = PFX(pixel_add_ps_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.add_psNONALIGNED = PFX(pixel_add_ps_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.add_psNONALIGNED   = PFX(pixel_add_ps_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.add_psNONALIGNED  = PFX(pixel_add_ps_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.add_psNONALIGNED = PFX(pixel_add_ps_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.add_psNONALIGNED = PFX(pixel_add_ps_32x64_neon);
-
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.add_psALIGNED   = PFX(pixel_add_ps_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.add_psALIGNED   = PFX(pixel_add_ps_8x8_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.add_psALIGNED = PFX(pixel_add_ps_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.add_psALIGNED = PFX(pixel_add_ps_32x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.add_psALIGNED   = PFX(pixel_add_ps_4x8_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.add_psALIGNED  = PFX(pixel_add_ps_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.add_psALIGNED = PFX(pixel_add_ps_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.add_psALIGNED = PFX(pixel_add_ps_32x64_neon);
-
     //scale2D_64to32
     p.scale2D_64to32  = PFX(scale2D_64to32_neon);
 
@@ -649,70 +497,8 @@
     p.scale1D_128to64NONALIGNED = PFX(scale1D_128to64_neon);
     p.scale1D_128to64ALIGNED = PFX(scale1D_128to64_neon);
 
-    // planecopy
-    p.planecopy_cp = PFX(pixel_planecopy_cp_neon);
-
-    // satd
-    ALL_LUMA_PU(satd, pixel_satd, neon);
-
-    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd   = PFX(pixel_satd_4x4_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_8x8.satd   = PFX(pixel_satd_8x8_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_16x16.satd = PFX(pixel_satd_16x16_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_32x32.satd = PFX(pixel_satd_32x32_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd   = PFX(pixel_satd_8x4_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_4x8.satd   = PFX(pixel_satd_4x8_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_16x8.satd  = PFX(pixel_satd_16x8_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_8x16.satd  = PFX(pixel_satd_8x16_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_32x16.satd = PFX(pixel_satd_32x16_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_16x32.satd = PFX(pixel_satd_16x32_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_16x12.satd = PFX(pixel_satd_16x12_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_12x16.satd = PFX(pixel_satd_12x16_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_16x4.satd  = PFX(pixel_satd_16x4_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_4x16.satd  = PFX(pixel_satd_4x16_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_32x24.satd = PFX(pixel_satd_32x24_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_24x32.satd = PFX(pixel_satd_24x32_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_32x8.satd  = PFX(pixel_satd_32x8_neon);
-    p.chromaX265_CSP_I420.puCHROMA_420_8x32.satd  = PFX(pixel_satd_8x32_neon);
-
-    p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd   = PFX(pixel_satd_4x8_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x16.satd  = PFX(pixel_satd_8x16_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_16x32.satd = PFX(pixel_satd_16x32_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_32x64.satd = PFX(pixel_satd_32x64_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd   = PFX(pixel_satd_4x4_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x8.satd   = PFX(pixel_satd_8x8_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_4x16.satd  = PFX(pixel_satd_4x16_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_16x16.satd = PFX(pixel_satd_16x16_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x32.satd  = PFX(pixel_satd_8x32_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_32x32.satd = PFX(pixel_satd_32x32_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_16x64.satd = PFX(pixel_satd_16x64_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd  = PFX(pixel_satd_8x12_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd   = PFX(pixel_satd_8x4_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_16x24.satd = PFX(pixel_satd_16x24_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd = PFX(pixel_satd_12x32_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_16x8.satd  = PFX(pixel_satd_16x8_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd  = PFX(pixel_satd_4x32_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_32x48.satd = PFX(pixel_satd_32x48_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_24x64.satd = PFX(pixel_satd_24x64_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_32x16.satd = PFX(pixel_satd_32x16_neon);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x64.satd  = PFX(pixel_satd_8x64_neon);
-
-    // sa8d
-    p.cuBLOCK_4x4.sa8d   = PFX(pixel_satd_4x4_neon);
-    p.cuBLOCK_8x8.sa8d   = PFX(pixel_sa8d_8x8_neon);
-    p.cuBLOCK_16x16.sa8d = PFX(pixel_sa8d_16x16_neon);
-    p.cuBLOCK_32x32.sa8d = PFX(pixel_sa8d_32x32_neon);
-    p.cuBLOCK_64x64.sa8d = PFX(pixel_sa8d_64x64_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_8x8.sa8d = PFX(pixel_satd_4x4_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_16x16.sa8d = PFX(pixel_sa8d_16x16_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_32x32.sa8d = PFX(pixel_sa8d_32x32_neon);
-    p.chromaX265_CSP_I420.cuBLOCK_64x64.sa8d = PFX(pixel_sa8d_64x64_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.sa8d = PFX(pixel_sa8d_8x16_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.sa8d = PFX(pixel_sa8d_16x32_neon);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sa8d = PFX(pixel_sa8d_32x64_neon);
-
     // dequant_scaling
     p.dequant_scaling = PFX(dequant_scaling_neon);
-    p.dequant_normal  = PFX(dequant_normal_neon);
 
     // ssim_4x4x2_core
     p.ssim_4x4x2_core = PFX(ssim_4x4x2_core_neon);
@@ -730,17 +516,13 @@
     p.cuBLOCK_32x32.normFact = PFX(normFact32_neon);
     p.cuBLOCK_64x64.normFact = PFX(normFact64_neon);
 
-    // psy_cost_pp
-    p.cuBLOCK_4x4.psy_cost_pp = PFX(psyCost_4x4_neon);
-
-    p.weight_pp = PFX(weight_pp_neon);
-#if !defined(__APPLE__)
     p.scanPosLast = PFX(scanPosLast_neon);
-#endif
+
     p.costCoeffNxN = PFX(costCoeffNxN_neon);
 #endif
 
     // quant
+    p.dequant_normal = PFX(dequant_normal_neon);
     p.quant = PFX(quant_neon);
     p.nquant = PFX(nquant_neon);
 }
@@ -757,149 +539,56 @@
     CHROMA_444_PU_SVE_FILTER_PIXEL_TO_SHORT(p2sNONALIGNED);
     LUMA_PU_SVE_FILTER_PIXEL_TO_SHORT(convert_p2sNONALIGNED);
 
-#if !HIGH_BIT_DEPTH
-    // Blockcopy_pp
-    LUMA_PU_CAN_USE_SVE_EXCEPT_FILTER_PIXEL_TO_SHORT(copy_pp, blockcopy_pp);
-    CHROMA_420_PU_CAN_USE_SVE_EXCEPT_FILTER_PIXEL_TO_SHORT(copy_pp, blockcopy_pp);
-    CHROMA_422_PU_CAN_USE_SVE_EXCEPT_FILTER_PIXEL_TO_SHORT(copy_pp, blockcopy_pp);
-    p.cuBLOCK_32x32.copy_pp = PFX(blockcopy_pp_32x32_sve);
-    p.cuBLOCK_64x64.copy_pp = PFX(blockcopy_pp_64x64_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_pp = PFX(blockcopy_pp_32x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_pp = PFX(blockcopy_pp_32x64_sve);
-
-#endif // !HIGH_BIT_DEPTH
-
-    // Blockcopy_ss
-    p.cuBLOCK_16x16.copy_ss = PFX(blockcopy_ss_16x16_sve);
-    p.cuBLOCK_32x32.copy_ss = PFX(blockcopy_ss_32x32_sve);
-    p.cuBLOCK_64x64.copy_ss = PFX(blockcopy_ss_64x64_sve);
-
-    // Blockcopy_ps
-    p.cuBLOCK_16x16.copy_ps = PFX(blockcopy_ps_16x16_sve);
-    p.cuBLOCK_32x32.copy_ps = PFX(blockcopy_ps_32x32_sve);
-    p.cuBLOCK_64x64.copy_ps = PFX(blockcopy_ps_64x64_sve);
-
-    // Blockcopy_sp
-    p.cuBLOCK_4x4.copy_sp   = PFX(blockcopy_sp_4x4_sve);
-    p.cuBLOCK_8x8.copy_sp   = PFX(blockcopy_sp_8x8_sve);
-    p.cuBLOCK_16x16.copy_sp = PFX(blockcopy_sp_16x16_sve);
-    p.cuBLOCK_32x32.copy_sp = PFX(blockcopy_sp_32x32_sve);
-
-    // chroma blockcopy_ss
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_ss = PFX(blockcopy_ss_16x16_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_ss = PFX(blockcopy_ss_32x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_ss = PFX(blockcopy_ss_16x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_ss = PFX(blockcopy_ss_32x64_sve);
-
-    // chroma blockcopy_ps
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_ps = PFX(blockcopy_ps_16x16_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_ps = PFX(blockcopy_ps_32x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_ps   = PFX(blockcopy_ps_4x8_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_ps  = PFX(blockcopy_ps_8x16_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_ps = PFX(blockcopy_ps_16x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_ps = PFX(blockcopy_ps_32x64_sve);
-
-    // chroma blockcopy_sp
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.copy_sp   = PFX(blockcopy_sp_4x4_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.copy_sp   = PFX(blockcopy_sp_8x8_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.copy_sp = PFX(blockcopy_sp_16x16_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.copy_sp = PFX(blockcopy_sp_32x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.copy_sp   = PFX(blockcopy_sp_4x8_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.copy_sp  = PFX(blockcopy_sp_8x16_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.copy_sp = PFX(blockcopy_sp_16x32_sve);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.copy_sp = PFX(blockcopy_sp_32x64_sve);
-
-    // Block_fill
-    LUMA_TU_CAN_USE_SVE(blockfill_sALIGNED, blockfill_s);
-    LUMA_TU_CAN_USE_SVE(blockfill_sNONALIGNED, blockfill_s);
-
-    // cpy2Dto1D_shl
-    p.cuBLOCK_16x16.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_16x16_sve);
-    p.cuBLOCK_32x32.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_32x32_sve);
-    p.cuBLOCK_64x64.cpy2Dto1D_shl = PFX(cpy2Dto1D_shl_64x64_sve);
-
-    // cpy2Dto1D_shr
-    p.cuBLOCK_16x16.cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_16x16_sve);
-    p.cuBLOCK_32x32.cpy2Dto1D_shr = PFX(cpy2Dto1D_shr_32x32_sve);
-
-    // cpy1Dto2D_shl
-    p.cuBLOCK_16x16.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_16x16_sve);
-    p.cuBLOCK_32x32.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_32x32_sve);
-    p.cuBLOCK_64x64.cpy1Dto2D_shlALIGNED    = PFX(cpy1Dto2D_shl_64x64_sve);
-
-    p.cuBLOCK_16x16.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_16x16_sve);
-    p.cuBLOCK_32x32.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_32x32_sve);
-    p.cuBLOCK_64x64.cpy1Dto2D_shlNONALIGNED = PFX(cpy1Dto2D_shl_64x64_sve);
+    // sse_ss
+    p.cuBLOCK_4x4.sse_ss   = PFX(pixel_sse_ss_4x4_sve);
+    p.cuBLOCK_8x8.sse_ss   = PFX(pixel_sse_ss_8x8_sve);
+    p.cuBLOCK_16x16.sse_ss = PFX(pixel_sse_ss_16x16_sve);
+    p.cuBLOCK_32x32.sse_ss = PFX(pixel_sse_ss_32x32_sve);
+    p.cuBLOCK_64x64.sse_ss = PFX(pixel_sse_ss_64x64_sve);
 
-    // cpy1Dto2D_shr
-    p.cuBLOCK_16x16.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_16x16_sve);
-    p.cuBLOCK_32x32.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_32x32_sve);
-    p.cuBLOCK_64x64.cpy1Dto2D_shr = PFX(cpy1Dto2D_shr_64x64_sve);
+    // ssd_s
+    p.cuBLOCK_4x4.ssd_sALIGNED      = PFX(pixel_ssd_s_4x4_sve);
+    p.cuBLOCK_8x8.ssd_sALIGNED      = PFX(pixel_ssd_s_8x8_sve);
+    p.cuBLOCK_16x16.ssd_sALIGNED    = PFX(pixel_ssd_s_16x16_sve);
+    p.cuBLOCK_32x32.ssd_sALIGNED    = PFX(pixel_ssd_s_32x32_sve);
+    p.cuBLOCK_64x64.ssd_sALIGNED    = PFX(pixel_ssd_s_64x64_sve);
+
+    p.cuBLOCK_4x4.ssd_sNONALIGNED   = PFX(pixel_ssd_s_4x4_sve);
+    p.cuBLOCK_8x8.ssd_sNONALIGNED   = PFX(pixel_ssd_s_8x8_sve);
+    p.cuBLOCK_16x16.ssd_sNONALIGNED = PFX(pixel_ssd_s_16x16_sve);
+    p.cuBLOCK_32x32.ssd_sNONALIGNED = PFX(pixel_ssd_s_32x32_sve);
+    p.cuBLOCK_64x64.ssd_sNONALIGNED = PFX(pixel_ssd_s_64x64_sve);
 
 #if !HIGH_BIT_DEPTH
     p.chromaX265_CSP_I422.cuBLOCK_422_8x16.sub_ps  = PFX(pixel_sub_ps_8x16_sve);
-
-    // satd
-    p.puLUMA_4x4.satd   = PFX(pixel_satd_4x4_sve);
-    p.puLUMA_32x32.satd = PFX(pixel_satd_32x32_sve);
-    p.puLUMA_8x4.satd   = PFX(pixel_satd_8x4_sve);
-    p.puLUMA_32x16.satd = PFX(pixel_satd_32x16_sve);
-    p.puLUMA_64x48.satd = PFX(pixel_satd_64x48_sve);
-
-    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd   = PFX(pixel_satd_4x4_sve);
-    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd   = PFX(pixel_satd_8x4_sve);
-
-    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd   = PFX(pixel_satd_4x4_sve);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd  = PFX(pixel_satd_8x12_sve);
-    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd   = PFX(pixel_satd_8x4_sve);
-
-    // sa8d
-    p.cuBLOCK_4x4.sa8d   = PFX(pixel_satd_4x4_sve);
-    p.chromaX265_CSP_I420.cuBLOCK_8x8.sa8d = PFX(pixel_satd_4x4_sve);
-#endif
+#else // HIGH_BIT_DEPTH
+    // sse_pp
+    p.cuBLOCK_4x4.sse_pp   = PFX(pixel_sse_pp_4x4_sve);
+    p.cuBLOCK_8x8.sse_pp   = PFX(pixel_sse_pp_8x8_sve);
+    p.cuBLOCK_16x16.sse_pp = PFX(pixel_sse_pp_16x16_sve);
+    p.cuBLOCK_32x32.sse_pp = PFX(pixel_sse_pp_32x32_sve);
+    p.cuBLOCK_64x64.sse_pp = PFX(pixel_sse_pp_64x64_sve);
+
+    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.sse_pp   = PFX(pixel_sse_pp_4x4_sve);
+    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.sse_pp   = PFX(pixel_sse_pp_8x8_sve);
+    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.sse_pp = PFX(pixel_sse_pp_16x16_sve);
+    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.sse_pp = PFX(pixel_sse_pp_32x32_sve);
+    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.sse_pp   = PFX(pixel_sse_pp_4x8_sve);
+    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.sse_pp  = PFX(pixel_sse_pp_8x16_sve);
+    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.sse_pp = PFX(pixel_sse_pp_16x32_sve);
+    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sse_pp = PFX(pixel_sse_pp_32x64_sve);
+#endif // !HIGH_BIT_DEPTH
 }
-#endif
+#endif // defined(HAVE_SVE2) || defined(HAVE_SVE)
 
 #if defined(HAVE_SVE2)
-#if !HIGH_BIT_DEPTH
 void setupSve2Primitives(EncoderPrimitives &p)
 {
+#if !HIGH_BIT_DEPTH
     // pixel_avg_pp
     LUMA_PU_MULTIPLE_ARCHS_3(pixelavg_ppNONALIGNED, pixel_avg_pp, sve2);
     LUMA_PU_MULTIPLE_ARCHS_3(pixelavg_ppALIGNED, pixel_avg_pp, sve2);
 
-    // addAvg
-    LUMA_PU_CAN_USE_SVE2(addAvgNONALIGNED, addAvg);
-    LUMA_PU_CAN_USE_SVE2(addAvgALIGNED, addAvg);
-    CHROMA_420_PU_MULTIPLE_ARCHS(addAvgNONALIGNED, addAvg, sve2);
-    CHROMA_420_PU_MULTIPLE_ARCHS(addAvgALIGNED, addAvg, sve2);
-    CHROMA_422_PU_CAN_USE_SVE2(addAvgNONALIGNED, addAvg);
-    CHROMA_422_PU_CAN_USE_SVE2(addAvgALIGNED, addAvg);
-
-    // sse_ss
-    p.cuBLOCK_4x4.sse_ss   = PFX(pixel_sse_ss_4x4_sve2);
-    p.cuBLOCK_8x8.sse_ss   = PFX(pixel_sse_ss_8x8_sve2);
-    p.cuBLOCK_16x16.sse_ss = PFX(pixel_sse_ss_16x16_sve2);
-    p.cuBLOCK_32x32.sse_ss = PFX(pixel_sse_ss_32x32_sve2);
-    p.cuBLOCK_64x64.sse_ss = PFX(pixel_sse_ss_64x64_sve2);
-
-    // ssd_s
-    p.cuBLOCK_4x4.ssd_sNONALIGNED   = PFX(pixel_ssd_s_4x4_sve2);
-    p.cuBLOCK_8x8.ssd_sNONALIGNED   = PFX(pixel_ssd_s_8x8_sve2);
-    p.cuBLOCK_16x16.ssd_sNONALIGNED = PFX(pixel_ssd_s_16x16_sve2);
-    p.cuBLOCK_32x32.ssd_sNONALIGNED = PFX(pixel_ssd_s_32x32_sve2);
-
-    p.cuBLOCK_4x4.ssd_sALIGNED   = PFX(pixel_ssd_s_4x4_sve2);
-    p.cuBLOCK_8x8.ssd_sALIGNED   = PFX(pixel_ssd_s_8x8_sve2);
-    p.cuBLOCK_16x16.ssd_sALIGNED = PFX(pixel_ssd_s_16x16_sve2);
-    p.cuBLOCK_32x32.ssd_sALIGNED = PFX(pixel_ssd_s_32x32_sve2);
-
-    // pixel_var
-    p.cuBLOCK_8x8.var   = PFX(pixel_var_8x8_sve2);
-    p.cuBLOCK_16x16.var = PFX(pixel_var_16x16_sve2);
-    p.cuBLOCK_32x32.var = PFX(pixel_var_32x32_sve2);
-    p.cuBLOCK_64x64.var = PFX(pixel_var_64x64_sve2);
-
     // calc_Residual
     p.cuBLOCK_16x16.calcresidualNONALIGNED = PFX(getResidual16_sve2);
     p.cuBLOCK_32x32.calcresidualNONALIGNED = PFX(getResidual32_sve2);
@@ -915,46 +604,10 @@
     p.chromaX265_CSP_I420.cuBLOCK_420_32x32.sub_ps = PFX(pixel_sub_ps_32x32_sve2);
     p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sub_ps = PFX(pixel_sub_ps_32x64_sve2);
 
-    // pixel_add_ps
-    p.cuBLOCK_4x4.add_psNONALIGNED   = PFX(pixel_add_ps_4x4_sve2);
-    p.cuBLOCK_8x8.add_psNONALIGNED   = PFX(pixel_add_ps_8x8_sve2);
-    p.cuBLOCK_16x16.add_psNONALIGNED = PFX(pixel_add_ps_16x16_sve2);
-    p.cuBLOCK_32x32.add_psNONALIGNED = PFX(pixel_add_ps_32x32_sve2);
-    p.cuBLOCK_64x64.add_psNONALIGNED = PFX(pixel_add_ps_64x64_sve2);
-
-    p.cuBLOCK_4x4.add_psALIGNED   = PFX(pixel_add_ps_4x4_sve2);
-    p.cuBLOCK_8x8.add_psALIGNED   = PFX(pixel_add_ps_8x8_sve2);
-    p.cuBLOCK_16x16.add_psALIGNED = PFX(pixel_add_ps_16x16_sve2);
-    p.cuBLOCK_32x32.add_psALIGNED = PFX(pixel_add_ps_32x32_sve2);
-    p.cuBLOCK_64x64.add_psALIGNED = PFX(pixel_add_ps_64x64_sve2);
-
-    // chroma add_ps
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.add_psNONALIGNED   = PFX(pixel_add_ps_4x4_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.add_psNONALIGNED   = PFX(pixel_add_ps_8x8_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.add_psNONALIGNED = PFX(pixel_add_ps_16x16_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.add_psNONALIGNED = PFX(pixel_add_ps_32x32_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.add_psNONALIGNED   = PFX(pixel_add_ps_4x8_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.add_psNONALIGNED  = PFX(pixel_add_ps_8x16_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.add_psNONALIGNED = PFX(pixel_add_ps_16x32_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.add_psNONALIGNED = PFX(pixel_add_ps_32x64_sve2);
-
-    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.add_psALIGNED   = PFX(pixel_add_ps_4x4_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.add_psALIGNED   = PFX(pixel_add_ps_8x8_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.add_psALIGNED = PFX(pixel_add_ps_16x16_sve2);
-    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.add_psALIGNED = PFX(pixel_add_ps_32x32_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.add_psALIGNED   = PFX(pixel_add_ps_4x8_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.add_psALIGNED  = PFX(pixel_add_ps_8x16_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.add_psALIGNED = PFX(pixel_add_ps_16x32_sve2);
-    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.add_psALIGNED = PFX(pixel_add_ps_32x64_sve2);
-
     // scale1D_128to64
     p.scale1D_128to64NONALIGNED = PFX(scale1D_128to64_sve2);
     p.scale1D_128to64ALIGNED = PFX(scale1D_128to64_sve2);
 
-    // dequant_scaling
-    p.dequant_scaling = PFX(dequant_scaling_sve2);
-    p.dequant_normal  = PFX(dequant_normal_sve2);
-
     // ssim_4x4x2_core
     p.ssim_4x4x2_core = PFX(ssim_4x4x2_core_sve2);
 
@@ -970,13 +623,18 @@
     p.cuBLOCK_16x16.normFact = PFX(normFact16_sve2);
     p.cuBLOCK_32x32.normFact = PFX(normFact32_sve2);
     p.cuBLOCK_64x64.normFact = PFX(normFact64_sve2);
+#endif // !HIGH_BIT_DEPTH
+
+    p.dequant_normal = PFX(dequant_normal_sve2);
 }
-#else // !HIGH_BIT_DEPTH
-void setupSve2Primitives(EncoderPrimitives &)
+#endif // defined(HAVE_SVE2)
+
+#if defined(HAVE_SVE2_BITPERM)
+void setupSve2BitPermPrimitives(EncoderPrimitives &p)
 {
+    p.scanPosLast = PFX(scanPosLast_sve2_bitperm);
 }
-#endif // !HIGH_BIT_DEPTH
-#endif // defined(HAVE_SVE2)
+#endif // defined(HAVE_SVE2_BITPERM)
 
 #ifdef HAVE_NEON_DOTPROD
 #if !HIGH_BIT_DEPTH
@@ -1028,6 +686,12 @@
         setupSve2Primitives(p);
     }
 #endif
+#ifdef HAVE_SVE2_BITPERM
+    if (cpuMask & X265_CPU_SVE2_BITPERM)
+    {
+        setupSve2BitPermPrimitives(p);
+    }
+#endif
 }
 
 void setupIntrinsicPrimitives(EncoderPrimitives &p, int cpuMask)
@@ -1045,6 +709,7 @@
     if (cpuMask & X265_CPU_NEON_DOTPROD)
     {
         setupFilterPrimitives_neon_dotprod(p);
+        setupPixelPrimitives_neon_dotprod(p);
     }
 #endif
 #ifdef HAVE_NEON_I8MM
@@ -1056,8 +721,10 @@
 #if defined(HAVE_SVE) && HAVE_SVE_BRIDGE
     if (cpuMask & X265_CPU_SVE)
     {
+        setupFilterPrimitives_sve(p);
         setupSaoPrimitives_sve(p);
         setupDCTPrimitives_sve(p);
+        setupPixelPrimitives_sve(p);
     }
 #endif
 #if defined(HAVE_SVE2) && HAVE_SVE_BRIDGE

x265-4.1.tar/source/common/aarch64/blockcopy8.S -> x265-4.2.tar/source/common/aarch64/blockcopy8.S Changed

@@ -22,7 +22,6 @@
  *****************************************************************************/
 
 #include "asm.S"
-#include "blockcopy8-common.S"
 
 #ifdef __APPLE__
 .section __RODATA,__rodata
@@ -34,773 +33,6 @@
 
 .text
 
-/* void blockcopy_sp(pixel* a, intptr_t stridea, const int16_t* b, intptr_t strideb)
- *
- * r0   - a
- * r1   - stridea
- * r2   - b
- * r3   - strideb */
-function PFX(blockcopy_sp_4x4_neon)
-    lsl             x3, x3, #1
-.rept 2
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    xtn             v0.8b, v0.8h
-    xtn             v1.8b, v1.8h
-    st1             {v0.s}0, x0, x1
-    st1             {v1.s}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_8x8_neon)
-    lsl             x3, x3, #1
-.rept 4
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    xtn             v0.8b, v0.8h
-    xtn             v1.8b, v1.8h
-    st1             {v0.d}0, x0, x1
-    st1             {v1.d}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_16x16_neon)
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.rept 8
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    st1             {v0.16b}, x0, x1
-    st1             {v1.16b}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_32x32_neon)
-    mov             w12, #4
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.Loop_csp32:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.8h-v3.8h}, x2, x3
-    ld1             {v4.8h-v7.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    tbl             v2.16b, {v4.16b,v5.16b}, v31.16b
-    tbl             v3.16b, {v6.16b,v7.16b}, v31.16b
-    st1             {v0.16b-v1.16b}, x0, x1
-    st1             {v2.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_csp32
-    ret
-endfunc
-
-function PFX(blockcopy_sp_64x64_neon)
-    mov             w12, #16
-    lsl             x3, x3, #1
-    sub             x3, x3, #64
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.Loop_csp64:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.8h-v3.8h}, x2, #64
-    ld1             {v4.8h-v7.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    tbl             v2.16b, {v4.16b,v5.16b}, v31.16b
-    tbl             v3.16b, {v6.16b,v7.16b}, v31.16b
-    st1             {v0.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_csp64
-    ret
-endfunc
-
-// void blockcopy_ps(int16_t* a, intptr_t stridea, const pixel* b, intptr_t strideb)
-function PFX(blockcopy_ps_4x4_neon)
-    lsl             x1, x1, #1
-.rept 2
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    st1             {v0.4h}, x0, x1
-    st1             {v1.4h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_8x8_neon)
-    lsl             x1, x1, #1
-.rept 4
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    st1             {v0.8h}, x0, x1
-    st1             {v1.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_16x16_neon)
-    lsl             x1, x1, #1
-.rept 8
-    ld1             {v4.16b}, x2, x3
-    ld1             {v5.16b}, x2, x3
-    uxtl            v0.8h, v4.8b
-    uxtl2           v1.8h, v4.16b
-    uxtl            v2.8h, v5.8b
-    uxtl2           v3.8h, v5.16b
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_32x32_neon)
-    lsl             x1, x1, #1
-    mov             w12, #4
-.Loop_cps32:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v17.16b}, x2, x3
-    ld1             {v18.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, x1
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps32
-    ret
-endfunc
-
-function PFX(blockcopy_ps_64x64_neon)
-    lsl             x1, x1, #1
-    sub             x1, x1, #64
-    mov             w12, #16
-.Loop_cps64:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, #64
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps64
-    ret
-endfunc
-
-// void x265_blockcopy_ss(int16_t* a, intptr_t stridea, const int16_t* b, intptr_t strideb)
-function PFX(blockcopy_ss_4x4_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 2
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    st1             {v0.8b}, x0, x1
-    st1             {v1.8b}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_8x8_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 4
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    st1             {v0.8h}, x0, x1
-    st1             {v1.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_16x16_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 8
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_32x32_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-    mov             w12, #4
-.Loop_css32:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css32
-    ret
-endfunc
-
-function PFX(blockcopy_ss_64x64_neon)
-    lsl             x1, x1, #1
-    sub             x1, x1, #64
-    lsl             x3, x3, #1
-    sub             x3, x3, #64
-    mov             w12, #8
-.Loop_css64:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, #64
-    ld1             {v4.8h-v7.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, #64
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css64
-    ret
-endfunc
-
-/******** Chroma blockcopy********/
-function PFX(blockcopy_ss_4x8_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 4
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    st1             {v0.8b}, x0, x1
-    st1             {v1.8b}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_8x16_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 8
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    st1             {v0.8h}, x0, x1
-    st1             {v1.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_16x32_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-.rept 16
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ss_32x64_neon)
-    lsl             x1, x1, #1
-    lsl             x3, x3, #1
-    mov             w12, #8
-.Loop_css32x64:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8h-v3.8h}, x2, x3
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_css32x64
-    ret
-endfunc
-
-// chroma blockcopy_ps
-function PFX(blockcopy_ps_4x8_neon)
-    lsl             x1, x1, #1
-.rept 4
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    st1             {v0.4h}, x0, x1
-    st1             {v1.4h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_8x16_neon)
-    lsl             x1, x1, #1
-.rept 8
-    ld1             {v0.8b}, x2, x3
-    ld1             {v1.8b}, x2, x3
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    st1             {v0.8h}, x0, x1
-    st1             {v1.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_16x32_neon)
-    lsl             x1, x1, #1
-.rept 16
-    ld1             {v4.16b}, x2, x3
-    ld1             {v5.16b}, x2, x3
-    uxtl            v0.8h, v4.8b
-    uxtl2           v1.8h, v4.16b
-    uxtl            v2.8h, v5.8b
-    uxtl2           v3.8h, v5.16b
-    st1             {v0.8h-v1.8h}, x0, x1
-    st1             {v2.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_ps_32x64_neon)
-    lsl             x1, x1, #1
-    mov             w12, #8
-.Loop_cps32x64:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v16.16b-v17.16b}, x2, x3
-    ld1             {v18.16b-v19.16b}, x2, x3
-    uxtl            v0.8h, v16.8b
-    uxtl2           v1.8h, v16.16b
-    uxtl            v2.8h, v17.8b
-    uxtl2           v3.8h, v17.16b
-    uxtl            v4.8h, v18.8b
-    uxtl2           v5.8h, v18.16b
-    uxtl            v6.8h, v19.8b
-    uxtl2           v7.8h, v19.16b
-    st1             {v0.8h-v3.8h}, x0, x1
-    st1             {v4.8h-v7.8h}, x0, x1
-.endr
-    cbnz            w12, .Loop_cps32x64
-    ret
-endfunc
-
-// chroma blockcopy_sp
-function PFX(blockcopy_sp_4x8_neon)
-    lsl             x3, x3, #1
-.rept 4
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    xtn             v0.8b, v0.8h
-    xtn             v1.8b, v1.8h
-    st1             {v0.s}0, x0, x1
-    st1             {v1.s}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_8x16_neon)
-    lsl             x3, x3, #1
-.rept 8
-    ld1             {v0.8h}, x2, x3
-    ld1             {v1.8h}, x2, x3
-    xtn             v0.8b, v0.8h
-    xtn             v1.8b, v1.8h
-    st1             {v0.d}0, x0, x1
-    st1             {v1.d}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_16x32_neon)
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.rept 16
-    ld1             {v0.8h-v1.8h}, x2, x3
-    ld1             {v2.8h-v3.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    st1             {v0.16b}, x0, x1
-    st1             {v1.16b}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_sp_32x64_neon)
-    mov             w12, #8
-    lsl             x3, x3, #1
-    movrel          x11, xtn_xtn2_table
-    ld1             {v31.16b}, x11
-.Loop_csp32x64:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.8h-v3.8h}, x2, x3
-    ld1             {v4.8h-v7.8h}, x2, x3
-    tbl             v0.16b, {v0.16b,v1.16b}, v31.16b
-    tbl             v1.16b, {v2.16b,v3.16b}, v31.16b
-    tbl             v2.16b, {v4.16b,v5.16b}, v31.16b
-    tbl             v3.16b, {v6.16b,v7.16b}, v31.16b
-    st1             {v0.16b-v1.16b}, x0, x1
-    st1             {v2.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_csp32x64
-    ret
-endfunc
-
-/* blockcopy_pp(pixel* dst, intptr_t dstStride, const pixel* src, intptr_t srcStride) */
-
-function PFX(blockcopy_pp_2x4_neon)
-    ldrh            w9, x2
-    add             x4, x1, x1
-    add             x14, x3, x3
-    strh            w9, x0
-    ldrh            w10, x2, x3
-    add             x5, x4, x1
-    add             x15, x14, x3
-    strh            w10, x0, x1
-    ldrh            w11, x2, x14
-    strh            w11, x0, x4
-    ldrh            w12, x2, x15
-    strh            w12, x0, x5
-    ret
-endfunc
-
-.macro blockcopy_pp_2xN_neon h
-function PFX(blockcopy_pp_2x\h\()_neon)
-    add             x4, x1, x1
-    add             x5, x4, x1
-    add             x6, x5, x1
-
-    add             x14, x3, x3
-    add             x15, x14, x3
-    add             x16, x15, x3
-
-.rept \h / 4
-    ldrh            w9, x2
-    strh            w9, x0
-    ldrh            w10, x2, x3
-    strh            w10, x0, x1
-    ldrh            w11, x2, x14
-    strh            w11, x0, x4
-    ldrh            w12, x2, x15
-    strh            w12, x0, x5
-    add             x2, x2, x16
-    add             x0, x0, x6
-.endr
-    ret
-endfunc
-.endm
-
-blockcopy_pp_2xN_neon 8
-blockcopy_pp_2xN_neon 16
-
-function PFX(blockcopy_pp_4x2_neon)
-    ldr             w9, x2
-    str             w9, x0
-    ldr             w10, x2, x3
-    str             w10, x0, x1
-    ret
-endfunc
-
-function PFX(blockcopy_pp_4x4_neon)
-    ldr             w9, x2
-    add             x4, x1, x1
-    add             x14, x3, x3
-    str             w9, x0
-    ldr             w10, x2, x3
-    add             x5, x4, x1
-    add             x15, x14, x3
-    str             w10, x0, x1
-    ldr             w11, x2, x14
-    str             w11, x0, x4
-    ldr             w12, x2, x15
-    str             w12, x0, x5
-    ret
-endfunc
-
-.macro blockcopy_pp_4xN_neon h
-function PFX(blockcopy_pp_4x\h\()_neon)
-    add             x4, x1, x1
-    add             x5, x4, x1
-    add             x6, x5, x1
-
-    add             x14, x3, x3
-    add             x15, x14, x3
-    add             x16, x15, x3
-
-.rept \h / 4
-    ldr             w9, x2
-    str             w9, x0
-    ldr             w10, x2, x3
-    str             w10, x0, x1
-    ldr             w11, x2, x14
-    str             w11, x0, x4
-    ldr             w12, x2, x15
-    str             w12, x0, x5
-    add             x2, x2, x16
-    add             x0, x0, x6
-.endr
-    ret
-endfunc
-.endm
-
-blockcopy_pp_4xN_neon 8
-blockcopy_pp_4xN_neon 16
-blockcopy_pp_4xN_neon 32
-
-.macro blockcopy_pp_6xN_neon h
-function PFX(blockcopy_pp_6x\h\()_neon)
-    sub             x1, x1, #4
-.rept \h
-    ld1             {v0.8b}, x2, x3
-    st1             {v0.s}0, x0, #4
-    st1             {v0.h}2, x0, x1
-.endr
-    ret
-endfunc
-.endm
-
-blockcopy_pp_6xN_neon 8
-blockcopy_pp_6xN_neon 16
-
-.macro blockcopy_pp_8xN_neon h
-function PFX(blockcopy_pp_8x\h\()_neon)
-.rept \h
-    ld1             {v0.4h}, x2, x3
-    st1             {v0.4h}, x0, x1
-.endr
-    ret
-endfunc
-.endm
-
-blockcopy_pp_8xN_neon 2
-blockcopy_pp_8xN_neon 4
-blockcopy_pp_8xN_neon 6
-blockcopy_pp_8xN_neon 8
-blockcopy_pp_8xN_neon 12
-blockcopy_pp_8xN_neon 16
-blockcopy_pp_8xN_neon 32
-
-function PFX(blockcopy_pp_8x64_neon)
-    mov             w12, #4
-.Loop_pp_8x64:
-    sub             w12, w12, #1
-.rept 16
-    ld1             {v0.4h}, x2, x3
-    st1             {v0.4h}, x0, x1
-.endr
-    cbnz            w12, .Loop_pp_8x64
-    ret
-endfunc
-
-.macro blockcopy_pp_16xN_neon h
-function PFX(blockcopy_pp_16x\h\()_neon)
-.rept \h
-    ld1             {v0.8h}, x2, x3
-    st1             {v0.8h}, x0, x1
-.endr
-    ret
-endfunc
-.endm
-
-blockcopy_pp_16xN_neon 4
-blockcopy_pp_16xN_neon 8
-blockcopy_pp_16xN_neon 12
-blockcopy_pp_16xN_neon 16
-
-.macro blockcopy_pp_16xN1_neon h
-function PFX(blockcopy_pp_16x\h\()_neon)
-    mov             w12, #\h / 8
-.Loop_16x\h\():
-.rept 8
-    ld1             {v0.8h}, x2, x3
-    st1             {v0.8h}, x0, x1
-.endr
-    sub             w12, w12, #1
-    cbnz            w12, .Loop_16x\h
-    ret
-endfunc
-.endm
-
-blockcopy_pp_16xN1_neon 24
-blockcopy_pp_16xN1_neon 32
-blockcopy_pp_16xN1_neon 64
-
-function PFX(blockcopy_pp_12x16_neon)
-    sub             x1, x1, #8
-.rept 16
-    ld1             {v0.16b}, x2, x3
-    str             d0, x0, #8
-    st1             {v0.s}2, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockcopy_pp_12x32_neon)
-    sub             x1, x1, #8
-    mov             w12, #4
-.Loop_pp_12x32:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.16b}, x2, x3
-    str             d0, x0, #8
-    st1             {v0.s}2, x0, x1
-.endr
-    cbnz            w12, .Loop_pp_12x32
-    ret
-endfunc
-
-function PFX(blockcopy_pp_24x32_neon)
-    mov             w12, #4
-.Loop_24x32:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.8b-v2.8b}, x2, x3
-    st1             {v0.8b-v2.8b}, x0, x1
-.endr
-    cbnz            w12, .Loop_24x32
-    ret
-endfunc
-
-function PFX(blockcopy_pp_24x64_neon)
-    mov             w12, #4
-.Loop_24x64:
-    sub             w12, w12, #1
-.rept 16
-    ld1             {v0.8b-v2.8b}, x2, x3
-    st1             {v0.8b-v2.8b}, x0, x1
-.endr
-    cbnz            w12, .Loop_24x64
-    ret
-endfunc
-
-function PFX(blockcopy_pp_32x8_neon)
-.rept 8
-    ld1             {v0.16b-v1.16b}, x2, x3
-    st1             {v0.16b-v1.16b}, x0, x1
-.endr
-    ret
-endfunc
-
-.macro blockcopy_pp_32xN_neon h
-function PFX(blockcopy_pp_32x\h\()_neon)
-    mov             w12, #\h / 8
-.Loop_32x\h\():
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.16b-v1.16b}, x2, x3
-    st1             {v0.16b-v1.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_32x\h
-    ret
-endfunc
-.endm
-
-blockcopy_pp_32xN_neon 16
-blockcopy_pp_32xN_neon 24
-blockcopy_pp_32xN_neon 32
-blockcopy_pp_32xN_neon 64
-blockcopy_pp_32xN_neon 48
-
-function PFX(blockcopy_pp_48x64_neon)
-    mov             w12, #8
-.Loop_48x64:
-    sub             w12, w12, #1
-.rept 8
-    ld1             {v0.16b-v2.16b}, x2, x3
-    st1             {v0.16b-v2.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_48x64
-    ret
-endfunc
-
-.macro blockcopy_pp_64xN_neon h
-function PFX(blockcopy_pp_64x\h\()_neon)
-    mov             w12, #\h / 4
-.Loop_64x\h\():
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.16b-v3.16b}, x2, x3
-    st1             {v0.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_64x\h
-    ret
-endfunc
-.endm
-
-blockcopy_pp_64xN_neon 16
-blockcopy_pp_64xN_neon 32
-blockcopy_pp_64xN_neon 48
-blockcopy_pp_64xN_neon 64
-
-// void x265_blockfill_s_neon(int16_t* dst, intptr_t dstride, int16_t val)
-function PFX(blockfill_s_4x4_neon)
-    dup             v0.4h, w2
-    lsl             x1, x1, #1
-.rept 4
-    st1             {v0.4h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockfill_s_8x8_neon)
-    dup             v0.8h, w2
-    lsl             x1, x1, #1
-.rept 8
-    st1             {v0.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockfill_s_16x16_neon)
-    dup             v0.8h, w2
-    mov             v1.16b, v0.16b
-    lsl             x1, x1, #1
-.rept 16
-    stp             q0, q1, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockfill_s_32x32_neon)
-    dup             v0.8h, w2
-    mov             v1.16b, v0.16b
-    mov             v2.16b, v0.16b
-    mov             v3.16b, v0.16b
-    lsl             x1, x1, #1
-.rept 32
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(blockfill_s_64x64_neon)
-    dup             v0.8h, w2
-    mov             v1.16b, v0.16b
-    mov             v2.16b, v0.16b
-    mov             v3.16b, v0.16b
-    lsl             x1, x1, #1
-    sub             x1, x1, #64
-.rept 64
-    st1             {v0.8h-v3.8h}, x0, #64
-    st1             {v0.8h-v3.8h}, x0, x1
-.endr
-    ret
-endfunc
-
 // uint32_t copy_count(int16_t* coeff, const int16_t* residual, intptr_t resiStride)
 function PFX(copy_cnt_4_neon)
     lsl             x2, x2, #1
@@ -960,340 +192,3 @@
     fmov            w0, s0
     ret
 endfunc
-
-// void cpy2Dto1D_shl(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
-.macro cpy2Dto1D_shl_start
-    add             x2, x2, x2
-    dup             v0.8h, w3
-.endm
-
-function PFX(cpy2Dto1D_shl_4x4_neon)
-    cpy2Dto1D_shl_start
-    ld1             {v2.d}0, x1, x2
-    ld1             {v2.d}1, x1, x2
-    ld1             {v3.d}0, x1, x2
-    ld1             {v3.d}1, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b-v3.16b}, x0
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_8x8_neon)
-    cpy2Dto1D_shl_start
-.rept 4
-    ld1             {v2.16b}, x1, x2
-    ld1             {v3.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b-v3.16b}, x0, #32
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_16x16_neon)
-    cpy2Dto1D_shl_start
-    mov             w12, #4
-.Loop_cpy2Dto1D_shl_16:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.16b-v3.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b-v3.16b}, x0, #32
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_16
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_32x32_neon)
-    cpy2Dto1D_shl_start
-    mov             w12, #16
-.Loop_cpy2Dto1D_shl_32:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_32
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shl_64x64_neon)
-    cpy2Dto1D_shl_start
-    mov             w12, #32
-    sub             x2, x2, #64
-.Loop_cpy2Dto1D_shl_64:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    ld1             {v16.16b-v19.16b}, x1, x2
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    sshl            v16.8h, v16.8h, v0.8h
-    sshl            v17.8h, v17.8h, v0.8h
-    sshl            v18.8h, v18.8h, v0.8h
-    sshl            v19.8h, v19.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-    st1             {v16.16b-v19.16b}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shl_64
-    ret
-endfunc
-
-// void cpy2Dto1D_shr(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
-function PFX(cpy2Dto1D_shr_4x4_neon)
-    cpy2Dto1D_shr_start
-    ld1             {v2.d}0, x1, x2
-    ld1             {v2.d}1, x1, x2
-    ld1             {v3.d}0, x1, x2
-    ld1             {v3.d}1, x1, x2
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    stp             q2, q3, x0
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_8x8_neon)
-    cpy2Dto1D_shr_start
-.rept 4
-    ld1             {v2.16b}, x1, x2
-    ld1             {v3.16b}, x1, x2
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    stp             q2, q3, x0, #32
-.endr
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_16x16_neon)
-    cpy2Dto1D_shr_start
-    mov             w12, #4
-.Loop_cpy2Dto1D_shr_16:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.8h-v3.8h}, x1, x2
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.8h-v3.8h}, x0, #32
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shr_16
-    ret
-endfunc
-
-function PFX(cpy2Dto1D_shr_32x32_neon)
-    cpy2Dto1D_shr_start
-    mov             w12, #16
-.Loop_cpy2Dto1D_shr_32:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.8h-v5.8h}, x1, x2
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sub             v4.8h, v4.8h, v1.8h
-    sub             v5.8h, v5.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.8h-v5.8h}, x0, #64
-.endr
-    cbnz            w12, .Loop_cpy2Dto1D_shr_32
-    ret
-endfunc
-
-// void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
-.macro cpy1Dto2D_shl_start
-    add             x2, x2, x2
-    dup             v0.8h, w3
-.endm
-
-function PFX(cpy1Dto2D_shl_4x4_neon)
-    cpy1Dto2D_shl_start
-    ld1             {v2.16b-v3.16b}, x1
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.d}0, x0, x2
-    st1             {v2.d}1, x0, x2
-    st1             {v3.d}0, x0, x2
-    st1             {v3.d}1, x0, x2
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_8x8_neon)
-    cpy1Dto2D_shl_start
-.rept 4
-    ld1             {v2.16b-v3.16b}, x1, #32
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b}, x0, x2
-    st1             {v3.16b}, x0, x2
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_16x16_neon)
-    cpy1Dto2D_shl_start
-    mov             w12, #4
-.Loop_cpy1Dto2D_shl_16:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.16b-v3.16b}, x1, #32
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b-v3.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shl_16
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_32x32_neon)
-    cpy1Dto2D_shl_start
-    mov             w12, #16
-.Loop_cpy1Dto2D_shl_32:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shl_32
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shl_64x64_neon)
-    cpy1Dto2D_shl_start
-    mov             w12, #32
-    sub             x2, x2, #64
-.Loop_cpy1Dto2D_shl_64:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    ld1             {v16.16b-v19.16b}, x1, #64
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    sshl            v16.8h, v16.8h, v0.8h
-    sshl            v17.8h, v17.8h, v0.8h
-    sshl            v18.8h, v18.8h, v0.8h
-    sshl            v19.8h, v19.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-    st1             {v16.16b-v19.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shl_64
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_4x4_neon)
-    cpy1Dto2D_shr_start
-    ld1             {v2.16b-v3.16b}, x1
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.d}0, x0, x2
-    st1             {v2.d}1, x0, x2
-    st1             {v3.d}0, x0, x2
-    st1             {v3.d}1, x0, x2
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_8x8_neon)
-    cpy1Dto2D_shr_start
-.rept 4
-    ld1             {v2.16b-v3.16b}, x1, #32
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.16b}, x0, x2
-    st1             {v3.16b}, x0, x2
-.endr
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_16x16_neon)
-    cpy1Dto2D_shr_start
-    mov             w12, #4
-.Loop_cpy1Dto2D_shr_16:
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v2.8h-v3.8h}, x1, #32
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    st1             {v2.8h-v3.8h}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shr_16
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_32x32_neon)
-    cpy1Dto2D_shr_start
-    mov             w12, #16
-.Loop_cpy1Dto2D_shr_32:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sub             v4.8h, v4.8h, v1.8h
-    sub             v5.8h, v5.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shr_32
-    ret
-endfunc
-
-function PFX(cpy1Dto2D_shr_64x64_neon)
-    cpy1Dto2D_shr_start
-    mov             w12, #32
-    sub             x2, x2, #64
-.Loop_cpy1Dto2D_shr_64:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v2.16b-v5.16b}, x1, #64
-    ld1             {v16.16b-v19.16b}, x1, #64
-    sub             v2.8h, v2.8h, v1.8h
-    sub             v3.8h, v3.8h, v1.8h
-    sub             v4.8h, v4.8h, v1.8h
-    sub             v5.8h, v5.8h, v1.8h
-    sub             v16.8h, v16.8h, v1.8h
-    sub             v17.8h, v17.8h, v1.8h
-    sub             v18.8h, v18.8h, v1.8h
-    sub             v19.8h, v19.8h, v1.8h
-    sshl            v2.8h, v2.8h, v0.8h
-    sshl            v3.8h, v3.8h, v0.8h
-    sshl            v4.8h, v4.8h, v0.8h
-    sshl            v5.8h, v5.8h, v0.8h
-    sshl            v16.8h, v16.8h, v0.8h
-    sshl            v17.8h, v17.8h, v0.8h
-    sshl            v18.8h, v18.8h, v0.8h
-    sshl            v19.8h, v19.8h, v0.8h
-    st1             {v2.16b-v5.16b}, x0, #64
-    st1             {v16.16b-v19.16b}, x0, x2
-.endr
-    cbnz            w12, .Loop_cpy1Dto2D_shr_64
-    ret
-endfunc

x265-4.1.tar/source/common/aarch64/cpu.h -> x265-4.2.tar/source/common/aarch64/cpu.h Changed

@@ -116,6 +116,14 @@
     }
 #endif  // defined(PF_ARM_SVE2_INSTRUCTIONS_AVAILABLE)
 #endif  // HAVE_SVE2
+#if HAVE_SVE2_BITPERM
+#if defined(PF_ARM_SVE_BITPERM_INSTRUCTIONS_AVAILABLE)
+    if (IsProcessorFeaturePresent(PF_ARM_SVE_BITPERM_INSTRUCTIONS_AVAILABLE))
+    {
+        flags |= X265_CPU_SVE2_BITPERM;
+    }
+#endif  // defined(PF_ARM_SVE_BITPERM_INSTRUCTIONS_AVAILABLE)
+#endif  // HAVE_SVE2_BITPERM
     return flags;
 }
 
@@ -126,6 +134,7 @@
 #define X265_AARCH64_HWCAP_ASIMDDP (1 << 20)
 #define X265_AARCH64_HWCAP_SVE (1 << 22)
 #define X265_AARCH64_HWCAP2_SVE2 (1 << 1)
+#define X265_AARCH64_HWCAP2_SVEBITPERM (1 << 4)
 #define X265_AARCH64_HWCAP2_I8MM (1 << 13)
 
 static inline int aarch64_get_cpu_flags()
@@ -135,7 +144,7 @@
 #if HAVE_NEON_DOTPROD || HAVE_SVE
     unsigned long hwcap = getauxval(AT_HWCAP);
 #endif
-#if HAVE_NEON_I8MM || HAVE_SVE2
+#if HAVE_NEON_I8MM || HAVE_SVE2 || HAVE_SVE2_BITPERM
     unsigned long hwcap2 = getauxval(AT_HWCAP2);
 #endif
 
@@ -154,6 +163,9 @@
 #if HAVE_SVE2
     if (hwcap2 & X265_AARCH64_HWCAP2_SVE2) flags |= X265_CPU_SVE2;
 #endif
+#if HAVE_SVE2_BITPERM
+    if (hwcap2 & X265_AARCH64_HWCAP2_SVEBITPERM) flags |= X265_CPU_SVE2_BITPERM;
+#endif
 
     return flags;
 }
@@ -179,6 +191,9 @@
     // Restrict flags: SVE2 assumes that FEAT_SVE is available.
     if (!(flags & X265_CPU_SVE)) flags &= ~X265_CPU_SVE2;
 
+    // Restrict flags: SVE2_BitPerm assumes that FEAT_SVE2 is available.
+    if (!(flags & X265_CPU_SVE2)) flags &= ~X265_CPU_SVE2_BITPERM;
+
     return flags;
 }
 
@@ -203,6 +218,9 @@
 #if HAVE_SVE2
     flags |= X265_CPU_SVE2;
 #endif
+#if HAVE_SVE2_BITPERM
+    flags |= X265_CPU_SVE2_BITPERM;
+#endif
     return flags;
 }

x265-4.1.tar/source/common/aarch64/dct-prim-sve.cpp -> x265-4.2.tar/source/common/aarch64/dct-prim-sve.cpp Changed

@@ -135,9 +135,9 @@
     }
 }
 
-template<int shift>
-static inline void partialButterfly16_sve(const int16_t *src, int16_t *dst)
+static inline void pass1Butterfly16_sve(const int16_t *src, int16_t *dst, intptr_t srcStride)
 {
+    const int shift = 3 + X265_DEPTH - 8;
     const int line = 16;
 
     int16x8_t Oline;
@@ -147,11 +147,11 @@
 
     for (int i = 0; i < line; i += 2)
     {
-        int16x8_t s0_lo = vld1q_s16(src + i * line);
-        int16x8_t s0_hi = rev16(vld1q_s16(src + i * line + 8));
+        int16x8_t s0_lo = vld1q_s16(src + i * srcStride);
+        int16x8_t s0_hi = rev16(vld1q_s16(src + i * srcStride + 8));
 
-        int16x8_t s1_lo = vld1q_s16(src + (i + 1) * line);
-        int16x8_t s1_hi = rev16(vld1q_s16(src + (i + 1) * line + 8));
+        int16x8_t s1_lo = vld1q_s16(src + (i + 1) * srcStride);
+        int16x8_t s1_hi = rev16(vld1q_s16(src + (i + 1) * srcStride + 8));
 
         int32x4_t E02;
         E00 = vaddl_s16(vget_low_s16(s0_lo), vget_low_s16(s0_hi));
@@ -239,9 +239,114 @@
     }
 }
 
-template<int shift>
-static inline void partialButterfly32_sve(const int16_t *src, int16_t *dst)
+static inline void pass2Butterfly16_sve(const int16_t *src, int16_t *dst)
+{
+    const int shift = 10;
+    const int line = 16;
+
+    int16x8_t Oline;
+    int32x4_t EOline;
+    int32x4_t EEEline;
+    int32x4_t EEOline;
+
+    for (int i = 0; i < line; i += 2)
+    {
+        int16x8_t s0_lo = vld1q_s16(src + i * line);
+        int16x8_t s0_hi = rev16(vld1q_s16(src + i * line + 8));
+
+        int16x8_t s1_lo = vld1q_s16(src + (i + 1) * line);
+        int16x8_t s1_hi = rev16(vld1q_s16(src + (i + 1) * line + 8));
+
+        int32x4_t E02;
+        E00 = vaddl_s16(vget_low_s16(s0_lo), vget_low_s16(s0_hi));
+        E01 = vaddl_s16(vget_high_s16(s0_lo), vget_high_s16(s0_hi));
+
+        int32x4_t E12;
+        E10 = vaddl_s16(vget_low_s16(s1_lo), vget_low_s16(s1_hi));
+        E11 = vaddl_s16(vget_high_s16(s1_lo), vget_high_s16(s1_hi));
+
+        Oi + 0 = vsubq_s16(s0_lo, s0_hi);
+        Oi + 1 = vsubq_s16(s1_lo, s1_hi);
+
+        EOi + 0 = vsubq_s32(E00, rev32(E01));
+        EOi + 1 = vsubq_s32(E10, rev32(E11));
+
+        int32x4_t EE0 = vaddq_s32(E00, rev32(E01));
+        int32x4_t EE1 = vaddq_s32(E10, rev32(E11));
+
+        int32x4_t t0 = vreinterpretq_s32_s64(
+            vzip1q_s64(vreinterpretq_s64_s32(EE0), vreinterpretq_s64_s32(EE1)));
+        int32x4_t t1 = vrev64q_s32(vreinterpretq_s32_s64(
+            vzip2q_s64(vreinterpretq_s64_s32(EE0),
+                       vreinterpretq_s64_s32(EE1))));
+
+        EEEi / 2 = vaddq_s32(t0, t1);
+        EEOi / 2 = vsubq_s32(t0, t1);
+    }
+
+    for (int i = 0; i < line; i += 4)
+    {
+        for (int k = 1; k < 16; k += 2)
+        {
+            int16x8_t c0_c4 = vld1q_s16(&g_t16k0);
+
+            int64x2_t t0 = x265_sdotq_s16(vdupq_n_s64(0), c0_c4, Oi + 0);
+            int64x2_t t1 = x265_sdotq_s16(vdupq_n_s64(0), c0_c4, Oi + 1);
+            int64x2_t t2 = x265_sdotq_s16(vdupq_n_s64(0), c0_c4, Oi + 2);
+            int64x2_t t3 = x265_sdotq_s16(vdupq_n_s64(0), c0_c4, Oi + 3);
+
+            int32x4_t t01 = vcombine_s32(vmovn_s64(t0), vmovn_s64(t1));
+            int32x4_t t23 = vcombine_s32(vmovn_s64(t2), vmovn_s64(t3));
+            int16x4_t res = vrshrn_n_s32(vpaddq_s32(t01, t23), shift);
+            vst1_s16(dst + k * line, res);
+        }
+
+        for (int k = 2; k < 16; k += 4)
+        {
+            int32x4_t c0 = x265_vld1sh_s32(&g_t16k0);
+
+            int32x4_t t0 = vmulq_s32(c0, EOi + 0);
+            int32x4_t t1 = vmulq_s32(c0, EOi + 1);
+            int32x4_t t2 = vmulq_s32(c0, EOi + 2);
+            int32x4_t t3 = vmulq_s32(c0, EOi + 3);
+            int32x4_t t = vpaddq_s32(vpaddq_s32(t0, t1), vpaddq_s32(t2, t3));
+
+            int16x4_t res = vrshrn_n_s32(t, shift);
+            vst1_s16(dst + k * line, res);
+        }
+
+        int32x4_t c0 = vld1q_s32(t8_even0);
+        int32x4_t c4 = vld1q_s32(t8_even1);
+        int32x4_t c8 = vld1q_s32(t8_even2);
+        int32x4_t c12 = vld1q_s32(t8_even3);
+
+        int32x4_t t0 = vpaddq_s32(EEEi / 2 + 0, EEEi / 2 + 1);
+        int32x4_t t1 = vmulq_s32(c0, t0);
+        int16x4_t res0 = vrshrn_n_s32(t1, shift);
+        vst1_s16(dst + 0 * line, res0);
+
+        int32x4_t t2 = vmulq_s32(c4, EEOi / 2 + 0);
+        int32x4_t t3 = vmulq_s32(c4, EEOi / 2 + 1);
+        int16x4_t res4 = vrshrn_n_s32(vpaddq_s32(t2, t3), shift);
+        vst1_s16(dst + 4 * line, res4);
+
+        int32x4_t t4 = vmulq_s32(c8, EEEi / 2 + 0);
+        int32x4_t t5 = vmulq_s32(c8, EEEi / 2 + 1);
+        int16x4_t res8 = vrshrn_n_s32(vpaddq_s32(t4, t5), shift);
+        vst1_s16(dst + 8 * line, res8);
+
+        int32x4_t t6 = vmulq_s32(c12, EEOi / 2 + 0);
+        int32x4_t t7 = vmulq_s32(c12, EEOi / 2 + 1);
+        int16x4_t res12 = vrshrn_n_s32(vpaddq_s32(t6, t7), shift);
+        vst1_s16(dst + 12 * line, res12);
+
+        dst += 4;
+    }
+}
+
+static inline void pass1Butterfly32_sve(const int16_t *src, int16_t *dst, intptr_t srcStride)
 {
+    const int shift = 4 + X265_DEPTH - 8;
     const int line = 32;
 
     int16x8_t Oline2;
@@ -252,11 +357,11 @@
 
     for (int i = 0; i < line; i += 2)
     {
-        int16x8x4_t in_lo = vld1q_s16_x4(src + (i + 0) * line);
+        int16x8x4_t in_lo = vld1q_s16_x4(src + (i + 0) * srcStride);
         in_lo.val2 = rev16(in_lo.val2);
         in_lo.val3 = rev16(in_lo.val3);
 
-        int16x8x4_t in_hi = vld1q_s16_x4(src + (i + 1) * line);
+        int16x8x4_t in_hi = vld1q_s16_x4(src + (i + 1) * srcStride);
         in_hi.val2 = rev16(in_hi.val2);
         in_hi.val3 = rev16(in_hi.val3);
 
@@ -424,6 +529,193 @@
     }
 }
 
+static inline void pass2Butterfly32_sve(const int16_t *src, int16_t *dst)
+{
+    const int shift = 11;
+    const int line = 32;
+
+    int16x8_t Oline2;
+    int32x4_t EOline2;
+    int32x4_t EEOline;
+    int32x4_t EEEEline / 2;
+    int32x4_t EEEOline / 2;
+
+    for (int i = 0; i < line; i += 2)
+    {
+        int16x8x4_t in_lo = vld1q_s16_x4(src + (i + 0) * line);
+        in_lo.val2 = rev16(in_lo.val2);
+        in_lo.val3 = rev16(in_lo.val3);
+
+        int16x8x4_t in_hi = vld1q_s16_x4(src + (i + 1) * line);
+        in_hi.val2 = rev16(in_hi.val2);
+        in_hi.val3 = rev16(in_hi.val3);
+
+        int32x4_t E04;
+        E00 = vaddl_s16(vget_low_s16(in_lo.val0),
+                          vget_low_s16(in_lo.val3));
+        E01 = vaddl_s16(vget_high_s16(in_lo.val0),
+                          vget_high_s16(in_lo.val3));
+        E02 = vaddl_s16(vget_low_s16(in_lo.val1),
+                          vget_low_s16(in_lo.val2));
+        E03 = vaddl_s16(vget_high_s16(in_lo.val1),
+                          vget_high_s16(in_lo.val2));
+
+        int32x4_t E14;
+        E10 = vaddl_s16(vget_low_s16(in_hi.val0),
+                          vget_low_s16(in_hi.val3));
+        E11 = vaddl_s16(vget_high_s16(in_hi.val0),
+                          vget_high_s16(in_hi.val3));
+        E12 = vaddl_s16(vget_low_s16(in_hi.val1),
+                          vget_low_s16(in_hi.val2));
+        E13 = vaddl_s16(vget_high_s16(in_hi.val1),
+                          vget_high_s16(in_hi.val2));
+
+        Oi + 00 = vsubq_s16(in_lo.val0, in_lo.val3);
+        Oi + 01 = vsubq_s16(in_lo.val1, in_lo.val2);
+
+        Oi + 10 = vsubq_s16(in_hi.val0, in_hi.val3);
+        Oi + 11 = vsubq_s16(in_hi.val1, in_hi.val2);
+
+        int32x4_t EE02;
+        E03 = rev32(E03);
+        E02 = rev32(E02);
+        EE00 = vaddq_s32(E00, E03);
+        EE01 = vaddq_s32(E01, E02);
+        EOi + 00 = vsubq_s32(E00, E03);
+        EOi + 01 = vsubq_s32(E01, E02);
+
+        int32x4_t EE12;
+        E13 = rev32(E13);
+        E12 = rev32(E12);
+        EE10 = vaddq_s32(E10, E13);
+        EE11 = vaddq_s32(E11, E12);
+        EOi + 10 = vsubq_s32(E10, E13);
+        EOi + 11 = vsubq_s32(E11, E12);
+
+        int32x4_t EEE0;
+        EE01 = rev32(EE01);
+        EEE0 = vaddq_s32(EE00, EE01);
+        EEOi + 0 = vsubq_s32(EE00, EE01);
+
+        int32x4_t EEE1;
+        EE11 = rev32(EE11);
+        EEE1 = vaddq_s32(EE10, EE11);
+        EEOi + 1 = vsubq_s32(EE10, EE11);
+
+        int32x4_t t0 = vreinterpretq_s32_s64(
+            vzip1q_s64(vreinterpretq_s64_s32(EEE0),
+                       vreinterpretq_s64_s32(EEE1)));
+        int32x4_t t1 = vrev64q_s32(vreinterpretq_s32_s64(
+            vzip2q_s64(vreinterpretq_s64_s32(EEE0),
+                       vreinterpretq_s64_s32(EEE1))));
+
+        EEEEi / 2 = vaddq_s32(t0, t1);
+        EEEOi / 2 = vsubq_s32(t0, t1);
+    }
+
+    for (int k = 1; k < 32; k += 2)
+    {
+        int16_t *d = dst + k * line;
+
+        int16x8_t c0_c1 = vld1q_s16(&g_t32k0);
+        int16x8_t c2_c3 = vld1q_s16(&g_t32k8);
+
+        for (int i = 0; i < line; i += 4)
+        {
+            int64x2_t t0 = x265_sdotq_s16(vdupq_n_s64(0), c0_c1, Oi + 00);
+            int64x2_t t1 = x265_sdotq_s16(vdupq_n_s64(0), c0_c1, Oi + 10);
+            int64x2_t t2 = x265_sdotq_s16(vdupq_n_s64(0), c0_c1, Oi + 20);
+            int64x2_t t3 = x265_sdotq_s16(vdupq_n_s64(0), c0_c1, Oi + 30);
+            t0 = x265_sdotq_s16(t0, c2_c3, Oi + 01);
+            t1 = x265_sdotq_s16(t1, c2_c3, Oi + 11);
+            t2 = x265_sdotq_s16(t2, c2_c3, Oi + 21);
+            t3 = x265_sdotq_s16(t3, c2_c3, Oi + 31);
+
+            int32x4_t t01 = vcombine_s32(vmovn_s64(t0), vmovn_s64(t1));
+            int32x4_t t23 = vcombine_s32(vmovn_s64(t2), vmovn_s64(t3));
+            int16x4_t res = vrshrn_n_s32(vpaddq_s32(t01, t23), shift);
+            vst1_s16(d, res);
+
+            d += 4;
+        }
+    }
+
+    for (int k = 2; k < 32; k += 4)
+    {
+        int16_t *d = dst + k * line;
+
+        int32x4_t c0 = x265_vld1sh_s32(&g_t32k0);
+        int32x4_t c1 = x265_vld1sh_s32(&g_t32k4);
+
+        for (int i = 0; i < line; i += 4)
+        {
+            int32x4_t t4;
+            for (int j = 0; j < 4; ++j) {
+                tj = vmulq_s32(c0, EOi + j0);
+                tj = vmlaq_s32(tj, c1, EOi + j1);
+            }
+
+            int32x4_t t0123 = vpaddq_s32(vpaddq_s32(t0, t1),
+                                         vpaddq_s32(t2, t3));
+            int16x4_t res = vrshrn_n_s32(t0123, shift);
+            vst1_s16(d, res);
+
+            d += 4;
+        }
+    }
+
+    for (int k = 4; k < 32; k += 8)
+    {
+        int16_t *d = dst + k * line;
+
+        int32x4_t c = x265_vld1sh_s32(&g_t32k0);
+
+        for (int i = 0; i < line; i += 4)
+        {
+            int32x4_t t0 = vmulq_s32(c, EEOi + 0);
+            int32x4_t t1 = vmulq_s32(c, EEOi + 1);
+            int32x4_t t2 = vmulq_s32(c, EEOi + 2);
+            int32x4_t t3 = vmulq_s32(c, EEOi + 3);
+
+            int32x4_t t = vpaddq_s32(vpaddq_s32(t0, t1), vpaddq_s32(t2, t3));
+            int16x4_t res = vrshrn_n_s32(t, shift);
+            vst1_s16(d, res);
+
+            d += 4;
+        }
+    }
+
+    int32x4_t c0 = vld1q_s32(t8_even0);
+    int32x4_t c8 = vld1q_s32(t8_even1);
+    int32x4_t c16 = vld1q_s32(t8_even2);
+    int32x4_t c24 = vld1q_s32(t8_even3);
+
+    for (int i = 0; i < line; i += 4)
+    {
+        int32x4_t t0 = vpaddq_s32(EEEEi / 2 + 0, EEEEi / 2 + 1);
+        int32x4_t t1 = vmulq_s32(c0, t0);
+        int16x4_t res0 = vrshrn_n_s32(t1, shift);
+        vst1_s16(dst + 0 * line, res0);
+
+        int32x4_t t2 = vmulq_s32(c8, EEEOi / 2 + 0);
+        int32x4_t t3 = vmulq_s32(c8, EEEOi / 2 + 1);
+        int16x4_t res8 = vrshrn_n_s32(vpaddq_s32(t2, t3), shift);
+        vst1_s16(dst + 8 * line, res8);
+
+        int32x4_t t4 = vmulq_s32(c16, EEEEi / 2 + 0);
+        int32x4_t t5 = vmulq_s32(c16, EEEEi / 2 + 1);
+        int16x4_t res16 = vrshrn_n_s32(vpaddq_s32(t4, t5), shift);
+        vst1_s16(dst + 16 * line, res16);
+
+        int32x4_t t6 = vmulq_s32(c24, EEEOi / 2 + 0);
+        int32x4_t t7 = vmulq_s32(c24, EEEOi / 2 + 1);
+        int16x4_t res24 = vrshrn_n_s32(vpaddq_s32(t6, t7), shift);
+        vst1_s16(dst + 24 * line, res24);
+
+        dst += 4;
+    }
+}
+
 }
 
 
@@ -449,36 +741,18 @@
 
 void dct16_sve(const int16_t *src, int16_t *dst, intptr_t srcStride)
 {
-    const int shift_pass1 = 3 + X265_DEPTH - 8;
-    const int shift_pass2 = 10;
-
     ALIGN_VAR_32(int16_t, coef16 * 16);
-    ALIGN_VAR_32(int16_t, block16 * 16);
-
-    for (int i = 0; i < 16; i++)
-    {
-        memcpy(&blocki * 16, &srci * srcStride, 16 * sizeof(int16_t));
-    }
 
-    partialButterfly16_sve<shift_pass1>(block, coef);
-    partialButterfly16_sve<shift_pass2>(coef, dst);
+    pass1Butterfly16_sve(src, coef, srcStride);
+    pass2Butterfly16_sve(coef, dst);
 }
 
 void dct32_sve(const int16_t *src, int16_t *dst, intptr_t srcStride)
 {
-    const int shift_pass1 = 4 + X265_DEPTH - 8;
-    const int shift_pass2 = 11;
-
     ALIGN_VAR_32(int16_t, coef32 * 32);
-    ALIGN_VAR_32(int16_t, block32 * 32);
-
-    for (int i = 0; i < 32; i++)
-    {
-        memcpy(&blocki * 32, &srci * srcStride, 32 * sizeof(int16_t));
-    }
 
-    partialButterfly32_sve<shift_pass1>(block, coef);
-    partialButterfly32_sve<shift_pass2>(coef, dst);
+    pass1Butterfly32_sve(src, coef, srcStride);
+    pass2Butterfly32_sve(coef, dst);
 }
 
 void setupDCTPrimitives_sve(EncoderPrimitives &p)

x265-4.1.tar/source/common/aarch64/dct-prim.cpp -> x265-4.2.tar/source/common/aarch64/dct-prim.cpp Changed

@@ -21,22 +21,50 @@
 {
 using namespace X265_NS;
 
-static void transpose_4x4x16(int16x4_t &x0, int16x4_t &x1, int16x4_t &x2, int16x4_t &x3)
+static inline void transpose_4x4_s16(int16x4_t &s0, int16x4_t &s1, int16x4_t &s2, int16x4_t &s3)
 {
-    int32x2_t s0, s1, s2, s3;
+    int16x8_t s0q = vcombine_s16(s0, vdup_n_s16(0));
+    int16x8_t s1q = vcombine_s16(s1, vdup_n_s16(0));
+    int16x8_t s2q = vcombine_s16(s2, vdup_n_s16(0));
+    int16x8_t s3q = vcombine_s16(s3, vdup_n_s16(0));
 
-    s0 = vtrn1_s32(vreinterpret_s32_s16(x0), vreinterpret_s32_s16(x2));
-    s1 = vtrn1_s32(vreinterpret_s32_s16(x1), vreinterpret_s32_s16(x3));
-    s2 = vtrn2_s32(vreinterpret_s32_s16(x0), vreinterpret_s32_s16(x2));
-    s3 = vtrn2_s32(vreinterpret_s32_s16(x1), vreinterpret_s32_s16(x3));
+    int16x8_t s02 = vzip1q_s16(s0q, s2q);
+    int16x8_t s13 = vzip1q_s16(s1q, s3q);
 
-    x0 = vtrn1_s16(vreinterpret_s16_s32(s0), vreinterpret_s16_s32(s1));
-    x1 = vtrn2_s16(vreinterpret_s16_s32(s0), vreinterpret_s16_s32(s1));
-    x2 = vtrn1_s16(vreinterpret_s16_s32(s2), vreinterpret_s16_s32(s3));
-    x3 = vtrn2_s16(vreinterpret_s16_s32(s2), vreinterpret_s16_s32(s3));
-}
+    int16x8x2_t s0123 = vzipq_s16(s02, s13);
 
+    s0 = vget_low_s16(s0123.val0);
+    s1 = vget_high_s16(s0123.val0);
+    s2 = vget_low_s16(s0123.val1);
+    s3 = vget_high_s16(s0123.val1);
+}
 
+static inline void transpose_4x8_s16(int16x4_t s0, int16x4_t s1, int16x4_t s2, int16x4_t s3,
+                                     int16x4_t s4, int16x4_t s5, int16x4_t s6, int16x4_t s7,
+                                     int16x8_t &d0, int16x8_t &d1, int16x8_t &d2, int16x8_t &d3)
+{
+    int16x8_t s0q = vcombine_s16(s0, vdup_n_s16(0));
+    int16x8_t s1q = vcombine_s16(s1, vdup_n_s16(0));
+    int16x8_t s2q = vcombine_s16(s2, vdup_n_s16(0));
+    int16x8_t s3q = vcombine_s16(s3, vdup_n_s16(0));
+    int16x8_t s4q = vcombine_s16(s4, vdup_n_s16(0));
+    int16x8_t s5q = vcombine_s16(s5, vdup_n_s16(0));
+    int16x8_t s6q = vcombine_s16(s6, vdup_n_s16(0));
+    int16x8_t s7q = vcombine_s16(s7, vdup_n_s16(0));
+
+    int16x8_t s04 = vzip1q_s16(s0q, s4q);
+    int16x8_t s15 = vzip1q_s16(s1q, s5q);
+    int16x8_t s26 = vzip1q_s16(s2q, s6q);
+    int16x8_t s37 = vzip1q_s16(s3q, s7q);
+
+    int16x8x2_t s0246 = vzipq_s16(s04, s26);
+    int16x8x2_t s1357 = vzipq_s16(s15, s37);
+
+    d0 = vzip1q_s16(s0246.val0, s1357.val0);
+    d1 = vzip2q_s16(s0246.val0, s1357.val0);
+    d2 = vzip1q_s16(s0246.val1, s1357.val1);
+    d3 = vzip2q_s16(s0246.val1, s1357.val1);
+}
 
 static int scanPosLast_opt(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign, uint16_t *coeffFlag,
                            uint8_t *coeffNum, int numSig, const uint16_t * /*scanCG4x4*/, const int /*trSize*/)
@@ -228,6 +256,123 @@
 }
 
 template<int shift>
+static inline void fastForwardDst4_neon(const int16_t *src, int16_t *dst)
+{
+    int16x4_t s0 = vld1_s16(src + 0);
+    int16x4_t s1 = vld1_s16(src + 4);
+    int16x4_t s2 = vld1_s16(src + 8);
+    int16x4_t s3 = vld1_s16(src + 12);
+
+    transpose_4x4_s16(s0, s1, s2, s3);
+
+    int32x4_t c0 = vaddl_s16(s0, s3);
+    int32x4_t c1 = vaddl_s16(s1, s3);
+    int32x4_t c2 = vsubl_s16(s0, s1);
+    int32x4_t c3 = vmull_n_s16(s2, 74);
+
+    int32x4_t t0 = vmlaq_n_s32(c3, c0, 29);
+    t0 = vmlaq_n_s32(t0, c1, 55);
+
+    int32x4_t t1 = vaddl_s16(s0, s1);
+    t1 = vsubw_s16(t1, s3);
+    t1 = vmulq_n_s32(t1, 74);
+
+    int32x4_t t2 = vmulq_n_s32(c2, 29);
+    t2 = vmlaq_n_s32(t2, c0, 55);
+    t2 = vsubq_s32(t2, c3);
+
+    int32x4_t t3 = vmlaq_n_s32(c3, c2, 55);
+    t3 = vmlsq_n_s32(t3, c1, 29);
+
+    int16x4_t d0 = vrshrn_n_s32(t0, shift);
+    int16x4_t d1 = vrshrn_n_s32(t1, shift);
+    int16x4_t d2 = vrshrn_n_s32(t2, shift);
+    int16x4_t d3 = vrshrn_n_s32(t3, shift);
+
+    vst1_s16(dst + 0, d0);
+    vst1_s16(dst + 4, d1);
+    vst1_s16(dst + 8, d2);
+    vst1_s16(dst + 12, d3);
+}
+
+template<int shift>
+static inline void inverseDst4_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
+{
+    int16x4_t s0 = vld1_s16(src + 0);
+    int16x4_t s1 = vld1_s16(src + 4);
+    int16x4_t s2 = vld1_s16(src + 8);
+    int16x4_t s3 = vld1_s16(src + 12);
+
+    int32x4_t c0 = vaddl_s16(s0, s2);
+    int32x4_t c1 = vaddl_s16(s2, s3);
+    int32x4_t c2 = vsubl_s16(s0, s3);
+    int32x4_t c3 = vmull_n_s16(s1, 74);
+
+    int32x4_t t0 = vmlaq_n_s32(c3, c0, 29);
+    t0 = vmlaq_n_s32(t0, c1, 55);
+
+    int32x4_t t1 = vmlaq_n_s32(c3, c2, 55);
+    t1 = vmlsq_n_s32(t1, c1, 29);
+
+    int32x4_t t2 = vaddl_s16(s0, s3);
+    t2 = vsubw_s16(t2, s2);
+    t2 = vmulq_n_s32(t2, 74);
+
+    int32x4_t t3 = vmulq_n_s32(c0, 55);
+    t3 = vmlaq_n_s32(t3, c2, 29);
+    t3 = vsubq_s32(t3, c3);
+
+    int16x4_t d0 = vqrshrn_n_s32(t0, shift);
+    int16x4_t d1 = vqrshrn_n_s32(t1, shift);
+    int16x4_t d2 = vqrshrn_n_s32(t2, shift);
+    int16x4_t d3 = vqrshrn_n_s32(t3, shift);
+
+    transpose_4x4_s16(d0, d1, d2, d3);
+
+    vst1_s16(dst + 0 * dstStride, d0);
+    vst1_s16(dst + 1 * dstStride, d1);
+    vst1_s16(dst + 2 * dstStride, d2);
+    vst1_s16(dst + 3 * dstStride, d3);
+}
+
+template<int shift>
+static inline void partialButterfly4_neon(const int16_t *src, int16_t *dst)
+{
+    int16x4_t s0 = vld1_s16(src + 0);
+    int16x4_t s1 = vld1_s16(src + 4);
+    int16x4_t s2 = vld1_s16(src + 8);
+    int16x4_t s3 = vld1_s16(src + 12);
+
+    transpose_4x4_s16(s0, s1, s2, s3);
+
+    int32x4_t E2, O2;
+    E0 = vaddl_s16(s0, s3);
+    O0 = vsubl_s16(s0, s3);
+    E1 = vaddl_s16(s1, s2);
+    O1 = vsubl_s16(s1, s2);
+
+    // Multiply and accumulate with g_t4 constants.
+    int32x4_t t0 = vaddq_s32(E0, E1);
+    t0 = vmulq_n_s32(t0, 64);
+    int32x4_t t1 = vmulq_n_s32(O0, 83);
+    t1 = vmlaq_n_s32(t1, O1, 36);
+    int32x4_t t2 = vsubq_s32(E0, E1);
+    t2 = vmulq_n_s32(t2, 64);
+    int32x4_t t3 = vmulq_n_s32(O0, 36);
+    t3 = vmlaq_n_s32(t3, O1, -83);
+
+    int16x4_t d0 = vrshrn_n_s32(t0, shift);
+    int16x4_t d1 = vrshrn_n_s32(t1, shift);
+    int16x4_t d2 = vrshrn_n_s32(t2, shift);
+    int16x4_t d3 = vrshrn_n_s32(t3, shift);
+
+    vst1_s16(dst + 0, d0);
+    vst1_s16(dst + 4, d1);
+    vst1_s16(dst + 8, d2);
+    vst1_s16(dst + 12, d3);
+}
+
+template<int shift>
 static inline void partialButterfly16_neon(const int16_t *src, int16_t *dst)
 {
     const int line = 16;
@@ -620,385 +765,997 @@
     }
 }
 
-static void partialButterflyInverse4(const int16_t *src, int16_t *dst, int shift, int line)
+template<int shift>
+static inline void partialButterflyInverse4_neon(const int16_t *src, int16_t *dst,
+                                                 intptr_t dstStride)
 {
-    int j;
-    int E2, O2;
-    int add = 1 << (shift - 1);
+    int16x4_t s0 = vld1_s16(src + 0);
+    int16x4_t s1 = vld1_s16(src + 4);
+    int16x4_t s2 = vld1_s16(src + 8);
+    int16x4_t s3 = vld1_s16(src + 12);
+
+    // Multiply and accumulate with g_t4 constants.
+    int32x4_t O2;
+    O0 = vmull_n_s16(s1, 83);
+    O0 = vmlal_n_s16(O0, s3, 36);
+    O1 = vmull_n_s16(s1, 36);
+    O1 = vmlal_n_s16(O1, s3, -83);
+
+    int32x4_t E2;
+    E0 = vaddl_s16(s0, s2);
+    E0 = vmulq_n_s32(E0, 64);
+    E1 = vsubl_s16(s0, s2);
+    E1 = vmulq_n_s32(E1, 64);
+
+    int32x4_t t0 = vaddq_s32(E0, O0);
+    int32x4_t t1 = vaddq_s32(E1, O1);
+    int32x4_t t2 = vsubq_s32(E1, O1);
+    int32x4_t t3 = vsubq_s32(E0, O0);
+
+    int16x4_t d0 = vqrshrn_n_s32(t0, shift);
+    int16x4_t d1 = vqrshrn_n_s32(t1, shift);
+    int16x4_t d2 = vqrshrn_n_s32(t2, shift);
+    int16x4_t d3 = vqrshrn_n_s32(t3, shift);
+
+    transpose_4x4_s16(d0, d1, d2, d3);
+
+    vst1_s16(dst + 0 * dstStride, d0);
+    vst1_s16(dst + 1 * dstStride, d1);
+    vst1_s16(dst + 2 * dstStride, d2);
+    vst1_s16(dst + 3 * dstStride, d3);
+}
+
+template<int shift>
+static inline void partialButterflyInverse8_neon(const int16_t *src, int16_t *dst,
+                                                 intptr_t dstStride)
+{
+    const int line = 8;
 
-    for (j = 0; j < line; j++)
+    const int16x8_t s0 = vld1q_s16(src + 0 * line);
+    const int16x8_t s1 = vld1q_s16(src + 1 * line);
+    const int16x8_t s2 = vld1q_s16(src + 2 * line);
+    const int16x8_t s3 = vld1q_s16(src + 3 * line);
+    const int16x8_t s4 = vld1q_s16(src + 4 * line);
+    const int16x8_t s5 = vld1q_s16(src + 5 * line);
+    const int16x8_t s6 = vld1q_s16(src + 6 * line);
+    const int16x8_t s7 = vld1q_s16(src + 7 * line);
+
+    int32x4_t O_lo4, O_hi4;
+    const int16x4_t c_odd = vld1_s16(g_t81);
+    O_lo0 = vmull_lane_s16(vget_low_s16(s1), c_odd, 0); // 89
+    O_lo1 = vmull_lane_s16(vget_low_s16(s1), c_odd, 1); // 75
+    O_lo2 = vmull_lane_s16(vget_low_s16(s1), c_odd, 2); // 50
+    O_lo3 = vmull_lane_s16(vget_low_s16(s1), c_odd, 3); // 18
+
+    O_hi0 = vmull_lane_s16(vget_high_s16(s1), c_odd, 0); // 89
+    O_hi1 = vmull_lane_s16(vget_high_s16(s1), c_odd, 1); // 75
+    O_hi2 = vmull_lane_s16(vget_high_s16(s1), c_odd, 2); // 50
+    O_hi3 = vmull_lane_s16(vget_high_s16(s1), c_odd, 3); // 18
+
+    if (vaddlvq_u32(vreinterpretq_u32_s16(s3)) != 0)
     {
-        /* Utilizing symmetry properties to the maximum to minimize the number of multiplications */
-        O0 = g_t410 * srcline + g_t430 * src3 * line;
-        O1 = g_t411 * srcline + g_t431 * src3 * line;
-        E0 = g_t400 * src0 + g_t420 * src2 * line;
-        E1 = g_t401 * src0 + g_t421 * src2 * line;
-
-        /* Combining even and odd terms at each hierarchy levels to calculate the final spatial domain vector */
-        dst0 = (int16_t)(x265_clip3(-32768, 32767, (E0 + O0 + add) >> shift));
-        dst1 = (int16_t)(x265_clip3(-32768, 32767, (E1 + O1 + add) >> shift));
-        dst2 = (int16_t)(x265_clip3(-32768, 32767, (E1 - O1 + add) >> shift));
-        dst3 = (int16_t)(x265_clip3(-32768, 32767, (E0 - O0 + add) >> shift));
-
-        src++;
-        dst += 4;
+        O_lo0 = vmlal_lane_s16(O_lo0, vget_low_s16(s3), c_odd, 1); //  75
+        O_lo1 = vmlsl_lane_s16(O_lo1, vget_low_s16(s3), c_odd, 3); // -18
+        O_lo2 = vmlsl_lane_s16(O_lo2, vget_low_s16(s3), c_odd, 0); // -89
+        O_lo3 = vmlsl_lane_s16(O_lo3, vget_low_s16(s3), c_odd, 2); // -50
+
+        O_hi0 = vmlal_lane_s16(O_hi0, vget_high_s16(s3), c_odd, 1); //  75
+        O_hi1 = vmlsl_lane_s16(O_hi1, vget_high_s16(s3), c_odd, 3); // -18
+        O_hi2 = vmlsl_lane_s16(O_hi2, vget_high_s16(s3), c_odd, 0); // -89
+        O_hi3 = vmlsl_lane_s16(O_hi3, vget_high_s16(s3), c_odd, 2); // -50
     }
-}
 
+    if (vaddlvq_u32(vreinterpretq_u32_s16(s5)) != 0)
+    {
+        O_lo0 = vmlal_lane_s16(O_lo0, vget_low_s16(s5), c_odd, 2); //  50
+        O_lo1 = vmlsl_lane_s16(O_lo1, vget_low_s16(s5), c_odd, 0); // -89
+        O_lo2 = vmlal_lane_s16(O_lo2, vget_low_s16(s5), c_odd, 3); //  18
+        O_lo3 = vmlal_lane_s16(O_lo3, vget_low_s16(s5), c_odd, 1); //  75
+
+        O_hi0 = vmlal_lane_s16(O_hi0, vget_high_s16(s5), c_odd, 2); //  50
+        O_hi1 = vmlsl_lane_s16(O_hi1, vget_high_s16(s5), c_odd, 0); // -89
+        O_hi2 = vmlal_lane_s16(O_hi2, vget_high_s16(s5), c_odd, 3); //  18
+        O_hi3 = vmlal_lane_s16(O_hi3, vget_high_s16(s5), c_odd, 1); //  75
+    }
 
+    if (vaddlvq_u32(vreinterpretq_u32_s16(s7)) != 0)
+    {
+        O_lo0 = vmlal_lane_s16(O_lo0, vget_low_s16(s7), c_odd, 3); //  18
+        O_lo1 = vmlsl_lane_s16(O_lo1, vget_low_s16(s7), c_odd, 2); // -50
+        O_lo2 = vmlal_lane_s16(O_lo2, vget_low_s16(s7), c_odd, 1); //  75
+        O_lo3 = vmlsl_lane_s16(O_lo3, vget_low_s16(s7), c_odd, 0); // -89
+
+        O_hi0 = vmlal_lane_s16(O_hi0, vget_high_s16(s7), c_odd, 3); //  18
+        O_hi1 = vmlsl_lane_s16(O_hi1, vget_high_s16(s7), c_odd, 2); // -50
+        O_hi2 = vmlal_lane_s16(O_hi2, vget_high_s16(s7), c_odd, 1); //  75
+        O_hi3 = vmlsl_lane_s16(O_hi3, vget_high_s16(s7), c_odd, 0); // -89
+    }
 
-static void partialButterflyInverse16_neon(const int16_t *src, int16_t *orig_dst, int shift, int line)
-{
-#define FMAK(x,l) sl = vmlal_lane_s16(sl,vld1_s16(&srcx*line),vld1_s16(&g_t16xk),l);
-#define MULK(x,l) vmull_lane_s16(vld1_s16(&srcx*line),vld1_s16(&g_t16xk),l);
-#define ODD3_15(k) FMAK(3,k);FMAK(5,k);FMAK(7,k);FMAK(9,k);FMAK(11,k);FMAK(13,k);FMAK(15,k);
-#define EVEN6_14_STEP4(k) FMAK(6,k);FMAK(10,k);FMAK(14,k);
+    int32x4_t EO_lo2, EO_hi2;
+    const int16x4_t c_even = vld1_s16(g_t82);
+    EO_lo0 = vmull_lane_s16(vget_low_s16(s2), c_even, 0); // 83
+    EO_lo1 = vmull_lane_s16(vget_low_s16(s2), c_even, 1); // 36
+
+    EO_hi0 = vmull_lane_s16(vget_high_s16(s2), c_even, 0); // 83
+    EO_hi1 = vmull_lane_s16(vget_high_s16(s2), c_even, 1); // 36
 
+    EO_lo0 = vmlal_lane_s16(EO_lo0, vget_low_s16(s6), c_even, 1); //  36
+    EO_lo1 = vmlsl_lane_s16(EO_lo1, vget_low_s16(s6), c_even, 0); // -83
 
-    int j, k;
-    int32x4_t E8, O8;
-    int32x4_t EE4, EO4;
-    int32x4_t EEE2, EEO2;
-    const int add = 1 << (shift - 1);
+    EO_hi0 = vmlal_lane_s16(EO_hi0, vget_high_s16(s6), c_even, 1); //  36
+    EO_hi1 = vmlsl_lane_s16(EO_hi1, vget_high_s16(s6), c_even, 0); // -83
 
+    // Replace multiply by 64 with left shift by 6.
+    int32x4_t EE_lo2, EE_hi2;
+    EE_lo0 = vshlq_n_s32(vaddl_s16(vget_low_s16(s0), vget_low_s16(s4)), 6);
+    EE_hi0 = vshlq_n_s32(vaddl_s16(vget_high_s16(s0), vget_high_s16(s4)), 6);
 
-X265_PRAGMA_UNROLL(4)
-    for (j = 0; j < line; j += 4)
+    EE_lo1 = vshll_n_s16(vget_low_s16(vsubq_s16(s0, s4)), 6);
+    EE_hi1 = vshll_n_s16(vget_high_s16(vsubq_s16(s0, s4)), 6);
+
+    int32x4_t E_lo4, E_hi4;
+    E_lo0 = vaddq_s32(EE_lo0, EO_lo0);
+    E_lo1 = vaddq_s32(EE_lo1, EO_lo1);
+    E_lo2 = vsubq_s32(EE_lo1, EO_lo1);
+    E_lo3 = vsubq_s32(EE_lo0, EO_lo0);
+
+    E_hi0 = vaddq_s32(EE_hi0, EO_hi0);
+    E_hi1 = vaddq_s32(EE_hi1, EO_hi1);
+    E_hi2 = vsubq_s32(EE_hi1, EO_hi1);
+    E_hi3 = vsubq_s32(EE_hi0, EO_hi0);
+
+    int16x4_t d_lo8, d_hi8;
+
+    for (int i = 0; i < 4; i++)
     {
-        /* Utilizing symmetry properties to the maximum to minimize the number of multiplications */
+        int32x4_t t_lo = vaddq_s32(E_loi, O_loi);
+        int32x4_t t_hi = vaddq_s32(E_hii, O_hii);
+        d_loi + 0 = vqrshrn_n_s32(t_lo, shift);
+        d_hii + 0 = vqrshrn_n_s32(t_hi, shift);
+
+        t_lo = vsubq_s32(E_lo3 - i, O_lo3 - i);
+        t_hi = vsubq_s32(E_hi3 - i, O_hi3 - i);
+        d_loi + 4 = vqrshrn_n_s32(t_lo, shift);
+        d_hii + 4 = vqrshrn_n_s32(t_hi, shift);
+    }
 
-X265_PRAGMA_UNROLL(2)
-        for (k = 0; k < 2; k++)
+    int16x8_t d0, d1, d2, d3, d4, d5, d6, d7;
+    transpose_4x8_s16(d_lo0, d_lo1, d_lo2, d_lo3, d_lo4, d_lo5, d_lo6, d_lo7,
+                      d0, d1, d2, d3);
+
+    transpose_4x8_s16(d_hi0, d_hi1, d_hi2, d_hi3, d_hi4, d_hi5, d_hi6, d_hi7,
+                      d4, d5, d6, d7);
+
+    vst1q_s16(dst + 0 * dstStride, d0);
+    vst1q_s16(dst + 1 * dstStride, d1);
+    vst1q_s16(dst + 2 * dstStride, d2);
+    vst1q_s16(dst + 3 * dstStride, d3);
+    vst1q_s16(dst + 4 * dstStride, d4);
+    vst1q_s16(dst + 5 * dstStride, d5);
+    vst1q_s16(dst + 6 * dstStride, d6);
+    vst1q_s16(dst + 7 * dstStride, d7);
+}
+
+template<int shift>
+static inline void partialButterflyInverse16_neon(const int16_t *src, int16_t *dst,
+                                                  intptr_t dstStride)
+{
+    const int line = 16;
+
+    for (int i = 0; i < 4; i++)
+    {
+        int32x4_t EEE2;
+        const int16x4_t s0 = vld1_s16(src + 0 * line + 4 * i);
+        const int16x4_t s8 = vld1_s16(src + 8 * line + 4 * i);
+        // Replace multiply by 64 with left shift by 6.
+        EEE0 = vshlq_n_s32(vaddl_s16(s0, s8), 6);
+        EEE1 = vshlq_n_s32(vsubl_s16(s0, s8), 6);
+
+        int32x4_t EEO2;
+        const int16x4_t c4_even = vld1_s16(g_t164);
+        const int16x4_t s4 = vld1_s16(src + 4 * line + 4 * i);
+        EEO0 = vmull_lane_s16(s4, c4_even, 0); // 83
+        EEO1 = vmull_lane_s16(s4, c4_even, 1); // 36
+
+        const int16x4_t s12 = vld1_s16(src + 12 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s12), 0) != 0)
         {
-            int32x4_t s;
-            s = vmull_s16(vdup_n_s16(g_t164k), vld1_s16(&src4 * line));
-            EEOk = vmlal_s16(s, vdup_n_s16(g_t1612k),
-                               vld1_s16(&src12 * line));
-            s = vmull_s16(vdup_n_s16(g_t160k), vld1_s16(&src0 * line));
-            EEEk = vmlal_s16(s, vdup_n_s16(g_t168k),
-                               vld1_s16(&src8 * line));
+            EEO0 = vmlal_lane_s16(EEO0, s12, c4_even, 1); //  36
+            EEO1 = vmlsl_lane_s16(EEO1, s12, c4_even, 0); // -83
         }
 
-        /* Combining even and odd terms at each hierarchy levels to calculate the final spatial domain vector */
-        EE0 = vaddq_s32(EEE0 , EEO0);
-        EE2 = vsubq_s32(EEE1 , EEO1);
-        EE1 = vaddq_s32(EEE1 , EEO1);
-        EE3 = vsubq_s32(EEE0 , EEO0);
+        int32x4_t EE4;
+        for (int j = 0; j < 2; j++)
+        {
+            EEj = vaddq_s32(EEEj, EEOj);
+            EEj + 2 = vsubq_s32(EEE1 - j, EEO1 - j);
+        }
 
+        int32x4_t EO4;
+        const int16x4_t c2_even = vld1_s16(g_t162);
+        const int16x4_t s2 = vld1_s16(src + 2 * line + 4 * i);
+        EO0 = vmull_lane_s16(s2, c2_even, 0); // 89
+        EO1 = vmull_lane_s16(s2, c2_even, 1); // 75
+        EO2 = vmull_lane_s16(s2, c2_even, 2); // 50
+        EO3 = vmull_lane_s16(s2, c2_even, 3); // 18
 
-X265_PRAGMA_UNROLL(1)
-        for (k = 0; k < 4; k += 4)
+        const int16x4_t s6 = vld1_s16(src + 6 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s6), 0) != 0)
         {
-            int32x4_t s4;
-            s0 = MULK(2, 0);
-            s1 = MULK(2, 1);
-            s2 = MULK(2, 2);
-            s3 = MULK(2, 3);
-
-            EVEN6_14_STEP4(0);
-            EVEN6_14_STEP4(1);
-            EVEN6_14_STEP4(2);
-            EVEN6_14_STEP4(3);
-
-            EOk = s0;
-            EOk + 1 = s1;
-            EOk + 2 = s2;
-            EOk + 3 = s3;
+            EO0 = vmlal_lane_s16(EO0, s6, c2_even, 1); //  75
+            EO1 = vmlsl_lane_s16(EO1, s6, c2_even, 3); // -18
+            EO2 = vmlsl_lane_s16(EO2, s6, c2_even, 0); // -89
+            EO3 = vmlsl_lane_s16(EO3, s6, c2_even, 2); // -50
         }
 
+        const int16x4_t s10 = vld1_s16(src + 10 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s10), 0) != 0)
+        {
+            EO0 = vmlal_lane_s16(EO0, s10, c2_even, 2); //  50
+            EO1 = vmlsl_lane_s16(EO1, s10, c2_even, 0); // -89
+            EO2 = vmlal_lane_s16(EO2, s10, c2_even, 3); //  18
+            EO3 = vmlal_lane_s16(EO3, s10, c2_even, 1); //  75
+        }
 
+        const int16x4_t s14 = vld1_s16(src + 14 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s14), 0) != 0)
+        {
+            EO0 = vmlal_lane_s16(EO0, s14, c2_even, 3); //  18
+            EO1 = vmlsl_lane_s16(EO1, s14, c2_even, 2); // -50
+            EO2 = vmlal_lane_s16(EO2, s14, c2_even, 1); //  75
+            EO3 = vmlsl_lane_s16(EO3, s14, c2_even, 0); // -89
+        }
 
-        static const int32x4_t min = vdupq_n_s32(-32768);
-        static const int32x4_t max = vdupq_n_s32(32767);
-        const int32x4_t minus_shift = vdupq_n_s32(-shift);
+        int32x4_t E8;
+        for (int j = 0; j < 4; j++)
+        {
+            Ej = vaddq_s32(EEj, EOj);
+            Ej + 4 = vsubq_s32(EE3 - j, EO3 - j);
+        }
 
-X265_PRAGMA_UNROLL(4)
-        for (k = 0; k < 4; k++)
+        int32x4_t O8;
+        const int16x8_t c_odd = vld1q_s16(g_t161);
+        const int16x4_t s1 = vld1_s16(src + 1 * line + 4 * i);
+        O0 = vmull_laneq_s16(s1, c_odd, 0); // 90
+        O1 = vmull_laneq_s16(s1, c_odd, 1); // 87
+        O2 = vmull_laneq_s16(s1, c_odd, 2); // 80
+        O3 = vmull_laneq_s16(s1, c_odd, 3); // 70
+        O4 = vmull_laneq_s16(s1, c_odd, 4); // 57
+        O5 = vmull_laneq_s16(s1, c_odd, 5); // 43
+        O6 = vmull_laneq_s16(s1, c_odd, 6); // 25
+        O7 = vmull_laneq_s16(s1, c_odd, 7); //  9
+
+        const int16x4_t s3 = vld1_s16(src + 3 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s3), 0) != 0)
         {
-            Ek = vaddq_s32(EEk , EOk);
-            Ek + 4 = vsubq_s32(EE3 - k , EO3 - k);
+            O0 = vmlal_laneq_s16(O0, s3, c_odd, 1); //  87
+            O1 = vmlal_laneq_s16(O1, s3, c_odd, 4); //  57
+            O2 = vmlal_laneq_s16(O2, s3, c_odd, 7); //   9
+            O3 = vmlsl_laneq_s16(O3, s3, c_odd, 5); // -43
+            O4 = vmlsl_laneq_s16(O4, s3, c_odd, 2); // -80
+            O5 = vmlsl_laneq_s16(O5, s3, c_odd, 0); // -90
+            O6 = vmlsl_laneq_s16(O6, s3, c_odd, 3); // -70
+            O7 = vmlsl_laneq_s16(O7, s3, c_odd, 6); // -25
         }
 
-X265_PRAGMA_UNROLL(2)
-        for (k = 0; k < 8; k += 4)
+        const int16x4_t s5 = vld1_s16(src + 5 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s5), 0) != 0)
         {
-            int32x4_t s4;
-            s0 = MULK(1, 0);
-            s1 = MULK(1, 1);
-            s2 = MULK(1, 2);
-            s3 = MULK(1, 3);
-            ODD3_15(0);
-            ODD3_15(1);
-            ODD3_15(2);
-            ODD3_15(3);
-            Ok = s0;
-            Ok + 1 = s1;
-            Ok + 2 = s2;
-            Ok + 3 = s3;
-            int32x4_t t;
-            int16x4_t x0, x1, x2, x3;
-
-            Ek = vaddq_s32(vdupq_n_s32(add), Ek);
-            t = vaddq_s32(Ek, Ok);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x0 = vmovn_s32(t);
-
-            Ek + 1 = vaddq_s32(vdupq_n_s32(add), Ek + 1);
-            t = vaddq_s32(Ek + 1, Ok + 1);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x1 = vmovn_s32(t);
-
-            Ek + 2 = vaddq_s32(vdupq_n_s32(add), Ek + 2);
-            t = vaddq_s32(Ek + 2, Ok + 2);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x2 = vmovn_s32(t);
-
-            Ek + 3 = vaddq_s32(vdupq_n_s32(add), Ek + 3);
-            t = vaddq_s32(Ek + 3, Ok + 3);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x3 = vmovn_s32(t);
-
-            transpose_4x4x16(x0, x1, x2, x3);
-            vst1_s16(&orig_dst0 * 16 + k, x0);
-            vst1_s16(&orig_dst1 * 16 + k, x1);
-            vst1_s16(&orig_dst2 * 16 + k, x2);
-            vst1_s16(&orig_dst3 * 16 + k, x3);
+            O0 = vmlal_laneq_s16(O0, s5, c_odd, 2); //  80
+            O1 = vmlal_laneq_s16(O1, s5, c_odd, 7); //   9
+            O2 = vmlsl_laneq_s16(O2, s5, c_odd, 3); // -70
+            O3 = vmlsl_laneq_s16(O3, s5, c_odd, 1); // -87
+            O4 = vmlsl_laneq_s16(O4, s5, c_odd, 6); // -25
+            O5 = vmlal_laneq_s16(O5, s5, c_odd, 4); //  57
+            O6 = vmlal_laneq_s16(O6, s5, c_odd, 0); //  90
+            O7 = vmlal_laneq_s16(O7, s5, c_odd, 5); //  43
         }
 
+        const int16x4_t s7 = vld1_s16(src + 7 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s7), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s7, c_odd, 3); //  70
+            O1 = vmlsl_laneq_s16(O1, s7, c_odd, 5); // -43
+            O2 = vmlsl_laneq_s16(O2, s7, c_odd, 1); // -87
+            O3 = vmlal_laneq_s16(O3, s7, c_odd, 7); //   9
+            O4 = vmlal_laneq_s16(O4, s7, c_odd, 0); //  90
+            O5 = vmlal_laneq_s16(O5, s7, c_odd, 6); //  25
+            O6 = vmlsl_laneq_s16(O6, s7, c_odd, 2); // -80
+            O7 = vmlsl_laneq_s16(O7, s7, c_odd, 4); // -57
+        }
 
-X265_PRAGMA_UNROLL(2)
-        for (k = 0; k < 8; k += 4)
+        const int16x4_t s9 = vld1_s16(src + 9 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s9), 0) != 0)
         {
-            int32x4_t t;
-            int16x4_t x0, x1, x2, x3;
-
-            t = vsubq_s32(E7 - k, O7 - k);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x0 = vmovn_s32(t);
-
-            t = vsubq_s32(E6 - k, O6 - k);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x1 = vmovn_s32(t);
-
-            t = vsubq_s32(E5 - k, O5 - k);
-
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x2 = vmovn_s32(t);
-
-            t = vsubq_s32(E4 - k, O4 - k);
-            t = vshlq_s32(t, minus_shift);
-            t = vmaxq_s32(t, min);
-            t = vminq_s32(t, max);
-            x3 = vmovn_s32(t);
-
-            transpose_4x4x16(x0, x1, x2, x3);
-            vst1_s16(&orig_dst0 * 16 + k + 8, x0);
-            vst1_s16(&orig_dst1 * 16 + k + 8, x1);
-            vst1_s16(&orig_dst2 * 16 + k + 8, x2);
-            vst1_s16(&orig_dst3 * 16 + k + 8, x3);
+            O0 = vmlal_laneq_s16(O0, s9, c_odd, 4); //  57
+            O1 = vmlsl_laneq_s16(O1, s9, c_odd, 2); // -80
+            O2 = vmlsl_laneq_s16(O2, s9, c_odd, 6); // -25
+            O3 = vmlal_laneq_s16(O3, s9, c_odd, 0); //  90
+            O4 = vmlsl_laneq_s16(O4, s9, c_odd, 7); //  -9
+            O5 = vmlsl_laneq_s16(O5, s9, c_odd, 1); // -87
+            O6 = vmlal_laneq_s16(O6, s9, c_odd, 5); //  43
+            O7 = vmlal_laneq_s16(O7, s9, c_odd, 3); //  70
         }
-        orig_dst += 4 * 16;
-        src += 4;
-    }
 
-#undef MUL
-#undef FMA
-#undef FMAK
-#undef MULK
-#undef ODD3_15
-#undef EVEN6_14_STEP4
+        const int16x4_t s11 = vld1_s16(src + 11 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s11), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s11, c_odd, 5); //  43
+            O1 = vmlsl_laneq_s16(O1, s11, c_odd, 0); // -90
+            O2 = vmlal_laneq_s16(O2, s11, c_odd, 4); //  57
+            O3 = vmlal_laneq_s16(O3, s11, c_odd, 6); //  25
+            O4 = vmlsl_laneq_s16(O4, s11, c_odd, 1); // -87
+            O5 = vmlal_laneq_s16(O5, s11, c_odd, 3); //  70
+            O6 = vmlal_laneq_s16(O6, s11, c_odd, 7); //   9
+            O7 = vmlsl_laneq_s16(O7, s11, c_odd, 2); // -80
+        }
 
+        const int16x4_t s13 = vld1_s16(src + 13 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s13), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s13, c_odd, 6); //  25
+            O1 = vmlsl_laneq_s16(O1, s13, c_odd, 3); // -70
+            O2 = vmlal_laneq_s16(O2, s13, c_odd, 0); //  90
+            O3 = vmlsl_laneq_s16(O3, s13, c_odd, 2); // -80
+            O4 = vmlal_laneq_s16(O4, s13, c_odd, 5); //  43
+            O5 = vmlal_laneq_s16(O5, s13, c_odd, 7); //   9
+            O6 = vmlsl_laneq_s16(O6, s13, c_odd, 4); // -57
+            O7 = vmlal_laneq_s16(O7, s13, c_odd, 1); //  87
+        }
 
-}
+        const int16x4_t s15 = vld1_s16(src + 15 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s15), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s15, c_odd, 7); //   9
+            O1 = vmlsl_laneq_s16(O1, s15, c_odd, 6); // -25
+            O2 = vmlal_laneq_s16(O2, s15, c_odd, 5); //  43
+            O3 = vmlsl_laneq_s16(O3, s15, c_odd, 4); // -57
+            O4 = vmlal_laneq_s16(O4, s15, c_odd, 3); //  70
+            O5 = vmlsl_laneq_s16(O5, s15, c_odd, 2); // -80
+            O6 = vmlal_laneq_s16(O6, s15, c_odd, 1); //  87
+            O7 = vmlsl_laneq_s16(O7, s15, c_odd, 0); // -90
+        }
 
+        int16x4_t d_lo8;
+        int16x4_t d_hi8;
+        for (int j = 0; j < 8; j++)
+        {
+            int32x4_t t_lo = vaddq_s32(Ej, Oj);
+            d_loj = vqrshrn_n_s32(t_lo, shift);
+
+            int32x4_t t_hi = vsubq_s32(E7 - j, O7 - j);
+            d_hij = vqrshrn_n_s32(t_hi, shift);
+        }
+
+        int16x8_t d0_lo, d1_lo, d2_lo, d3_lo;
+        int16x8_t d0_hi, d1_hi, d2_hi, d3_hi;
+        transpose_4x8_s16(d_lo0, d_lo1, d_lo2, d_lo3, d_lo4, d_lo5, d_lo6, d_lo7,
+                          d0_lo, d1_lo, d2_lo, d3_lo);
+        transpose_4x8_s16(d_hi0, d_hi1, d_hi2, d_hi3, d_hi4, d_hi5, d_hi6, d_hi7,
+                          d0_hi, d1_hi, d2_hi, d3_hi);
 
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 0, d0_lo);
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 1, d0_hi);
 
-static void partialButterflyInverse32_neon(const int16_t *src, int16_t *orig_dst, int shift, int line)
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 0, d1_lo);
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 1, d1_hi);
+
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 0, d2_lo);
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 1, d2_hi);
+
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 0, d3_lo);
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 1, d3_hi);
+    }
+}
+
+template<int shift>
+static inline void partialButterflyInverse32_neon(const int16_t *src, int16_t *dst,
+                                                  intptr_t dstStride)
 {
-#define MUL(x) vmull_s16(vdup_n_s16(g_t32xk),vld1_s16(&srcx*line));
-#define FMA(x) s = vmlal_s16(s,vdup_n_s16(g_t32xk),vld1_s16(&srcx*line));
-#define FMAK(x,l) sl = vmlal_lane_s16(sl,vld1_s16(&srcx*line),vld1_s16(&g_t32xk),l);
-#define MULK(x,l) vmull_lane_s16(vld1_s16(&srcx*line),vld1_s16(&g_t32xk),l);
-#define ODD31(k) FMAK(3,k);FMAK(5,k);FMAK(7,k);FMAK(9,k);FMAK(11,k);FMAK(13,k);FMAK(15,k);FMAK(17,k);FMAK(19,k);FMAK(21,k);FMAK(23,k);FMAK(25,k);FMAK(27,k);FMAK(29,k);FMAK(31,k);
-
-#define ODD15(k) FMAK(6,k);FMAK(10,k);FMAK(14,k);FMAK(18,k);FMAK(22,k);FMAK(26,k);FMAK(30,k);
-#define ODD7(k) FMAK(12,k);FMAK(20,k);FMAK(28,k);
-
-
-    int j, k;
-    int32x4_t E16, O16;
-    int32x4_t EE8, EO8;
-    int32x4_t EEE4, EEO4;
-    int32x4_t EEEE2, EEEO2;
-    int16x4_t dst32;
-    int add = 1 << (shift - 1);
-
-X265_PRAGMA_UNROLL(8)
-    for (j = 0; j < line; j += 4)
+    const int line = 32;
+
+    for (int i = 0; i < 8; i++)
     {
-X265_PRAGMA_UNROLL(4)
-        for (k = 0; k < 16; k += 4)
+        int32x4_t EEEE2;
+        const int16x4_t s0 = vld1_s16(src + 0 * line + 4 * i);
+        const int16x4_t s16 = vld1_s16(src + 16 * line + 4 * i);
+        // Replace multiply by 64 with left shift by 6.
+        EEEE0 = vshlq_n_s32(vaddl_s16(s0, s16), 6);
+        EEEE1 = vshlq_n_s32(vsubl_s16(s0, s16), 6);
+
+        int32x4_t EEEO2;
+        const int16x4_t c8 = vld1_s16(g_t328);
+        const int16x4_t s8 = vld1_s16(src + 8 * line + 4 * i);
+        EEEO0 = vmull_lane_s16(s8, c8, 0); // 83
+        EEEO1 = vmull_lane_s16(s8, c8, 1); // 36
+
+        const int16x4_t s24 = vld1_s16(src + 24 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s24), 0) != 0)
+        {
+            EEEO0 = vmlal_lane_s16(EEEO0, s24, c8, 1); //  36
+            EEEO1 = vmlsl_lane_s16(EEEO1, s24, c8, 0); // -83
+        }
+
+        int32x4_t EEE4;
+        for (int j = 0; j < 2; j++)
+        {
+            EEEj = vaddq_s32(EEEEj, EEEOj);
+            EEEj + 2 = vsubq_s32(EEEE1 - j, EEEO1 - j);
+        }
+
+        int32x4_t EEO4;
+        const int16x4_t c4 = vld1_s16(g_t324);
+        const int16x4_t s4 = vld1_s16(src + 4 * line + 4 * i);
+        EEO0 = vmull_lane_s16(s4, c4, 0); // 89
+        EEO1 = vmull_lane_s16(s4, c4, 1); // 75
+        EEO2 = vmull_lane_s16(s4, c4, 2); // 50
+        EEO3 = vmull_lane_s16(s4, c4, 3); // 18
+
+        const int16x4_t s12 = vld1_s16(src + 12 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s12), 0) != 0)
+        {
+            EEO0 = vmlal_lane_s16(EEO0, s12, c4, 1); //  75
+            EEO1 = vmlsl_lane_s16(EEO1, s12, c4, 3); // -18
+            EEO2 = vmlsl_lane_s16(EEO2, s12, c4, 0); // -89
+            EEO3 = vmlsl_lane_s16(EEO3, s12, c4, 2); // -50
+        }
+
+        const int16x4_t s20 = vld1_s16(src + 20 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s20), 0) != 0)
+        {
+            EEO0 = vmlal_lane_s16(EEO0, s20, c4, 2); //  50
+            EEO1 = vmlsl_lane_s16(EEO1, s20, c4, 0); // -89
+            EEO2 = vmlal_lane_s16(EEO2, s20, c4, 3); //  18
+            EEO3 = vmlal_lane_s16(EEO3, s20, c4, 1); //  75
+        }
+
+        const int16x4_t s28 = vld1_s16(src + 28 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s28), 0) != 0)
+        {
+            EEO0 = vmlal_lane_s16(EEO0, s28, c4, 3); //  18
+            EEO1 = vmlsl_lane_s16(EEO1, s28, c4, 2); // -50
+            EEO2 = vmlal_lane_s16(EEO2, s28, c4, 1); //  75
+            EEO3 = vmlsl_lane_s16(EEO3, s28, c4, 0); // -89
+        }
+
+        int32x4_t EE8;
+        for (int j = 0; j < 4; j++)
+        {
+            EEj = vaddq_s32(EEEj, EEOj);
+            EEj + 4 = vsubq_s32(EEE3 - j, EEO3 - j);
+        }
+
+        int32x4_t EO8;
+        const int16x8_t c2 = vld1q_s16(g_t322);
+        const int16x4_t s2 = vld1_s16(src + 2 * line + 4 * i);
+        EO0 = vmull_laneq_s16(s2, c2, 0); // 90
+        EO1 = vmull_laneq_s16(s2, c2, 1); // 87
+        EO2 = vmull_laneq_s16(s2, c2, 2); // 80
+        EO3 = vmull_laneq_s16(s2, c2, 3); // 70
+        EO4 = vmull_laneq_s16(s2, c2, 4); // 57
+        EO5 = vmull_laneq_s16(s2, c2, 5); // 43
+        EO6 = vmull_laneq_s16(s2, c2, 6); // 25
+        EO7 = vmull_laneq_s16(s2, c2, 7); //  9
+
+        const int16x4_t s6 = vld1_s16(src + 6 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s6), 0) != 0)
+        {
+            EO0 = vmlal_laneq_s16(EO0, s6, c2, 1); //  87
+            EO1 = vmlal_laneq_s16(EO1, s6, c2, 4); //  57
+            EO2 = vmlal_laneq_s16(EO2, s6, c2, 7); //   9
+            EO3 = vmlsl_laneq_s16(EO3, s6, c2, 5); // -43
+            EO4 = vmlsl_laneq_s16(EO4, s6, c2, 2); // -80
+            EO5 = vmlsl_laneq_s16(EO5, s6, c2, 0); // -90
+            EO6 = vmlsl_laneq_s16(EO6, s6, c2, 3); // -70
+            EO7 = vmlsl_laneq_s16(EO7, s6, c2, 6); // -25
+        }
+
+        const int16x4_t s10 = vld1_s16(src + 10 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s10), 0) != 0)
+        {
+            EO0 = vmlal_laneq_s16(EO0, s10, c2, 2); //  80
+            EO1 = vmlal_laneq_s16(EO1, s10, c2, 7); //   9
+            EO2 = vmlsl_laneq_s16(EO2, s10, c2, 3); // -70
+            EO3 = vmlsl_laneq_s16(EO3, s10, c2, 1); // -87
+            EO4 = vmlsl_laneq_s16(EO4, s10, c2, 6); // -25
+            EO5 = vmlal_laneq_s16(EO5, s10, c2, 4); //  57
+            EO6 = vmlal_laneq_s16(EO6, s10, c2, 0); //  90
+            EO7 = vmlal_laneq_s16(EO7, s10, c2, 5); //  43
+        }
+
+        const int16x4_t s14 = vld1_s16(src + 14 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s14), 0) != 0)
         {
-            int32x4_t s4;
-            s0 = MULK(1, 0);
-            s1 = MULK(1, 1);
-            s2 = MULK(1, 2);
-            s3 = MULK(1, 3);
-            ODD31(0);
-            ODD31(1);
-            ODD31(2);
-            ODD31(3);
-            Ok = s0;
-            Ok + 1 = s1;
-            Ok + 2 = s2;
-            Ok + 3 = s3;
+            EO0 = vmlal_laneq_s16(EO0, s14, c2, 3); //  70
+            EO1 = vmlsl_laneq_s16(EO1, s14, c2, 5); // -43
+            EO2 = vmlsl_laneq_s16(EO2, s14, c2, 1); // -87
+            EO3 = vmlal_laneq_s16(EO3, s14, c2, 7); //   9
+            EO4 = vmlal_laneq_s16(EO4, s14, c2, 0); //  90
+            EO5 = vmlal_laneq_s16(EO5, s14, c2, 6); //  25
+            EO6 = vmlsl_laneq_s16(EO6, s14, c2, 2); // -80
+            EO7 = vmlsl_laneq_s16(EO7, s14, c2, 4); // -57
+        }
 
+        const int16x4_t s18 = vld1_s16(src + 18 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s18), 0) != 0)
+        {
+            EO0 = vmlal_laneq_s16(EO0, s18, c2, 4); //  57
+            EO1 = vmlsl_laneq_s16(EO1, s18, c2, 2); // -80
+            EO2 = vmlsl_laneq_s16(EO2, s18, c2, 6); // -25
+            EO3 = vmlal_laneq_s16(EO3, s18, c2, 0); //  90
+            EO4 = vmlsl_laneq_s16(EO4, s18, c2, 7); //  -9
+            EO5 = vmlsl_laneq_s16(EO5, s18, c2, 1); // -87
+            EO6 = vmlal_laneq_s16(EO6, s18, c2, 5); //  43
+            EO7 = vmlal_laneq_s16(EO7, s18, c2, 3); //  70
+        }
 
+        const int16x4_t s22 = vld1_s16(src + 22 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s22), 0) != 0)
+        {
+            EO0 = vmlal_laneq_s16(EO0, s22, c2, 5); //  43
+            EO1 = vmlsl_laneq_s16(EO1, s22, c2, 0); // -90
+            EO2 = vmlal_laneq_s16(EO2, s22, c2, 4); //  57
+            EO3 = vmlal_laneq_s16(EO3, s22, c2, 6); //  25
+            EO4 = vmlsl_laneq_s16(EO4, s22, c2, 1); // -87
+            EO5 = vmlal_laneq_s16(EO5, s22, c2, 3); //  70
+            EO6 = vmlal_laneq_s16(EO6, s22, c2, 7); //   9
+            EO7 = vmlsl_laneq_s16(EO7, s22, c2, 2); // -80
         }
 
+        const int16x4_t s26 = vld1_s16(src + 26 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s26), 0) != 0)
+        {
+            EO0 = vmlal_laneq_s16(EO0, s26, c2, 6); //  25
+            EO1 = vmlsl_laneq_s16(EO1, s26, c2, 3); // -70
+            EO2 = vmlal_laneq_s16(EO2, s26, c2, 0); //  90
+            EO3 = vmlsl_laneq_s16(EO3, s26, c2, 2); // -80
+            EO4 = vmlal_laneq_s16(EO4, s26, c2, 5); //  43
+            EO5 = vmlal_laneq_s16(EO5, s26, c2, 7); //   9
+            EO6 = vmlsl_laneq_s16(EO6, s26, c2, 4); // -57
+            EO7 = vmlal_laneq_s16(EO7, s26, c2, 1); //  87
+        }
 
-X265_PRAGMA_UNROLL(2)
-        for (k = 0; k < 8; k += 4)
+        const int16x4_t s30 = vld1_s16(src + 30 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s30), 0) != 0)
         {
-            int32x4_t s4;
-            s0 = MULK(2, 0);
-            s1 = MULK(2, 1);
-            s2 = MULK(2, 2);
-            s3 = MULK(2, 3);
-
-            ODD15(0);
-            ODD15(1);
-            ODD15(2);
-            ODD15(3);
-
-            EOk = s0;
-            EOk + 1 = s1;
-            EOk + 2 = s2;
-            EOk + 3 = s3;
+            EO0 = vmlal_laneq_s16(EO0, s30, c2, 7); //   9
+            EO1 = vmlsl_laneq_s16(EO1, s30, c2, 6); // -25
+            EO2 = vmlal_laneq_s16(EO2, s30, c2, 5); //  43
+            EO3 = vmlsl_laneq_s16(EO3, s30, c2, 4); // -57
+            EO4 = vmlal_laneq_s16(EO4, s30, c2, 3); //  70
+            EO5 = vmlsl_laneq_s16(EO5, s30, c2, 2); // -80
+            EO6 = vmlal_laneq_s16(EO6, s30, c2, 1); //  87
+            EO7 = vmlsl_laneq_s16(EO7, s30, c2, 0); // -90
         }
 
+        int32x4_t E16;
+        for (int j = 0; j < 8; j++)
+        {
+            Ej = vaddq_s32(EEj, EOj);
+            Ej + 8 = vsubq_s32(EE7 - j, EO7 - j);
+        }
+
+        int32x4_t O16;
+        const int16x8_t c1_lo = vld1q_s16(g_t321 + 0);
+        const int16x8_t c1_hi = vld1q_s16(g_t321 + 8);
+        const int16x4_t s1 = vld1_s16(src + 1 * line + 4 * i);
+        O0 = vmull_laneq_s16(s1, c1_lo, 0);  // 90
+        O1 = vmull_laneq_s16(s1, c1_lo, 1);  // 90
+        O2 = vmull_laneq_s16(s1, c1_lo, 2);  // 88
+        O3 = vmull_laneq_s16(s1, c1_lo, 3);  // 85
+        O4 = vmull_laneq_s16(s1, c1_lo, 4);  // 82
+        O5 = vmull_laneq_s16(s1, c1_lo, 5);  // 78
+        O6 = vmull_laneq_s16(s1, c1_lo, 6);  // 73
+        O7 = vmull_laneq_s16(s1, c1_lo, 7);  // 67
+        O8 = vmull_laneq_s16(s1, c1_hi, 0);  // 61
+        O9 = vmull_laneq_s16(s1, c1_hi, 1);  // 54
+        O10 = vmull_laneq_s16(s1, c1_hi, 2); // 46
+        O11 = vmull_laneq_s16(s1, c1_hi, 3); // 38
+        O12 = vmull_laneq_s16(s1, c1_hi, 4); // 31
+        O13 = vmull_laneq_s16(s1, c1_hi, 5); // 22
+        O14 = vmull_laneq_s16(s1, c1_hi, 6); // 13
+        O15 = vmull_laneq_s16(s1, c1_hi, 7); //  4
+
+        const int16x4_t s3 = vld1_s16(src + 3 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s3), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s3, c1_lo, 1);   //  90
+            O1 = vmlal_laneq_s16(O1, s3, c1_lo, 4);   //  82
+            O2 = vmlal_laneq_s16(O2, s3, c1_lo, 7);   //  67
+            O3 = vmlal_laneq_s16(O3, s3, c1_hi, 2);   //  46
+            O4 = vmlal_laneq_s16(O4, s3, c1_hi, 5);   //  22
+            O5 = vmlsl_laneq_s16(O5, s3, c1_hi, 7);   //  -4
+            O6 = vmlsl_laneq_s16(O6, s3, c1_hi, 4);   // -31
+            O7 = vmlsl_laneq_s16(O7, s3, c1_hi, 1);   // -54
+            O8 = vmlsl_laneq_s16(O8, s3, c1_lo, 6);   // -73
+            O9 = vmlsl_laneq_s16(O9, s3, c1_lo, 3);   // -85
+            O10 = vmlsl_laneq_s16(O10, s3, c1_lo, 0); // -90
+            O11 = vmlsl_laneq_s16(O11, s3, c1_lo, 2); // -88
+            O12 = vmlsl_laneq_s16(O12, s3, c1_lo, 5); // -78
+            O13 = vmlsl_laneq_s16(O13, s3, c1_hi, 0); // -61
+            O14 = vmlsl_laneq_s16(O14, s3, c1_hi, 3); // -38
+            O15 = vmlsl_laneq_s16(O15, s3, c1_hi, 6); // -13
+        }
 
-        for (k = 0; k < 4; k += 4)
+        const int16x4_t s5 = vld1_s16(src + 5 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s5), 0) != 0)
         {
-            int32x4_t s4;
-            s0 = MULK(4, 0);
-            s1 = MULK(4, 1);
-            s2 = MULK(4, 2);
-            s3 = MULK(4, 3);
-
-            ODD7(0);
-            ODD7(1);
-            ODD7(2);
-            ODD7(3);
-
-            EEOk = s0;
-            EEOk + 1 = s1;
-            EEOk + 2 = s2;
-            EEOk + 3 = s3;
+            O0 = vmlal_laneq_s16(O0, s5, c1_lo, 2);   //  88
+            O1 = vmlal_laneq_s16(O1, s5, c1_lo, 7);   //  67
+            O2 = vmlal_laneq_s16(O2, s5, c1_hi, 4);   //  31
+            O3 = vmlsl_laneq_s16(O3, s5, c1_hi, 6);   // -13
+            O4 = vmlsl_laneq_s16(O4, s5, c1_hi, 1);   // -54
+            O5 = vmlsl_laneq_s16(O5, s5, c1_lo, 4);   // -82
+            O6 = vmlsl_laneq_s16(O6, s5, c1_lo, 0);   // -90
+            O7 = vmlsl_laneq_s16(O7, s5, c1_lo, 5);   // -78
+            O8 = vmlsl_laneq_s16(O8, s5, c1_hi, 2);   // -46
+            O9 = vmlsl_laneq_s16(O9, s5, c1_hi, 7);   //  -4
+            O10 = vmlal_laneq_s16(O10, s5, c1_hi, 3); //  38
+            O11 = vmlal_laneq_s16(O11, s5, c1_lo, 6); //  73
+            O12 = vmlal_laneq_s16(O12, s5, c1_lo, 1); //  90
+            O13 = vmlal_laneq_s16(O13, s5, c1_lo, 3); //  85
+            O14 = vmlal_laneq_s16(O14, s5, c1_hi, 0); //  61
+            O15 = vmlal_laneq_s16(O15, s5, c1_hi, 5); //  22
         }
 
-X265_PRAGMA_UNROLL(2)
-        for (k = 0; k < 2; k++)
+        const int16x4_t s7 = vld1_s16(src + 7 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s7), 0) != 0)
         {
-            int32x4_t s;
-            s = MUL(8);
-            EEEOk = FMA(24);
-            s = MUL(0);
-            EEEEk = FMA(16);
+            O0 = vmlal_laneq_s16(O0, s7, c1_lo, 3);   //  85
+            O1 = vmlal_laneq_s16(O1, s7, c1_hi, 2);   //  46
+            O2 = vmlsl_laneq_s16(O2, s7, c1_hi, 6);   // -13
+            O3 = vmlsl_laneq_s16(O3, s7, c1_lo, 7);   // -67
+            O4 = vmlsl_laneq_s16(O4, s7, c1_lo, 0);   // -90
+            O5 = vmlsl_laneq_s16(O5, s7, c1_lo, 6);   // -73
+            O6 = vmlsl_laneq_s16(O6, s7, c1_hi, 5);   // -22
+            O7 = vmlal_laneq_s16(O7, s7, c1_hi, 3);   //  38
+            O8 = vmlal_laneq_s16(O8, s7, c1_lo, 4);   //  82
+            O9 = vmlal_laneq_s16(O9, s7, c1_lo, 2);   //  88
+            O10 = vmlal_laneq_s16(O10, s7, c1_hi, 1); //  54
+            O11 = vmlsl_laneq_s16(O11, s7, c1_hi, 7); //  -4
+            O12 = vmlsl_laneq_s16(O12, s7, c1_hi, 0); // -61
+            O13 = vmlsl_laneq_s16(O13, s7, c1_lo, 1); // -90
+            O14 = vmlsl_laneq_s16(O14, s7, c1_lo, 5); // -78
+            O15 = vmlsl_laneq_s16(O15, s7, c1_hi, 4); // -31
         }
-        /* Combining even and odd terms at each hierarchy levels to calculate the final spatial domain vector */
-        EEE0 = vaddq_s32(EEEE0, EEEO0);
-        EEE3 = vsubq_s32(EEEE0, EEEO0);
-        EEE1 = vaddq_s32(EEEE1, EEEO1);
-        EEE2 = vsubq_s32(EEEE1, EEEO1);
-
-X265_PRAGMA_UNROLL(4)
-        for (k = 0; k < 4; k++)
+
+        const int16x4_t s9 = vld1_s16(src + 9 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s9), 0) != 0)
         {
-            EEk = vaddq_s32(EEEk, EEOk);
-            EEk + 4 = vsubq_s32((EEE3 - k), (EEO3 - k));
+            O0 = vmlal_laneq_s16(O0, s9, c1_lo, 4);   //  82
+            O1 = vmlal_laneq_s16(O1, s9, c1_hi, 5);   //  22
+            O2 = vmlsl_laneq_s16(O2, s9, c1_hi, 1);   // -54
+            O3 = vmlsl_laneq_s16(O3, s9, c1_lo, 0);   // -90
+            O4 = vmlsl_laneq_s16(O4, s9, c1_hi, 0);   // -61
+            O5 = vmlal_laneq_s16(O5, s9, c1_hi, 6);   //  13
+            O6 = vmlal_laneq_s16(O6, s9, c1_lo, 5);   //  78
+            O7 = vmlal_laneq_s16(O7, s9, c1_lo, 3);   //  85
+            O8 = vmlal_laneq_s16(O8, s9, c1_hi, 4);   //  31
+            O9 = vmlsl_laneq_s16(O9, s9, c1_hi, 2);   // -46
+            O10 = vmlsl_laneq_s16(O10, s9, c1_lo, 1); // -90
+            O11 = vmlsl_laneq_s16(O11, s9, c1_lo, 7); // -67
+            O12 = vmlal_laneq_s16(O12, s9, c1_hi, 7); //   4
+            O13 = vmlal_laneq_s16(O13, s9, c1_lo, 6); //  73
+            O14 = vmlal_laneq_s16(O14, s9, c1_lo, 2); //  88
+            O15 = vmlal_laneq_s16(O15, s9, c1_hi, 3); //  38
         }
 
-X265_PRAGMA_UNROLL(8)
-        for (k = 0; k < 8; k++)
+        const int16x4_t s11 = vld1_s16(src + 11 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s11), 0) != 0)
         {
-            Ek = vaddq_s32(EEk, EOk);
-            Ek + 8 = vsubq_s32((EE7 - k), (EO7 - k));
+            O0 = vmlal_laneq_s16(O0, s11, c1_lo, 5);   //  78
+            O1 = vmlsl_laneq_s16(O1, s11, c1_hi, 7);   //  -4
+            O2 = vmlsl_laneq_s16(O2, s11, c1_lo, 4);   // -82
+            O3 = vmlsl_laneq_s16(O3, s11, c1_lo, 6);   // -73
+            O4 = vmlal_laneq_s16(O4, s11, c1_hi, 6);   //  13
+            O5 = vmlal_laneq_s16(O5, s11, c1_lo, 3);   //  85
+            O6 = vmlal_laneq_s16(O6, s11, c1_lo, 7);   //  67
+            O7 = vmlsl_laneq_s16(O7, s11, c1_hi, 5);   // -22
+            O8 = vmlsl_laneq_s16(O8, s11, c1_lo, 2);   // -88
+            O9 = vmlsl_laneq_s16(O9, s11, c1_hi, 0);   // -61
+            O10 = vmlal_laneq_s16(O10, s11, c1_hi, 4); //  31
+            O11 = vmlal_laneq_s16(O11, s11, c1_lo, 1); //  90
+            O12 = vmlal_laneq_s16(O12, s11, c1_hi, 1); //  54
+            O13 = vmlsl_laneq_s16(O13, s11, c1_hi, 3); // -38
+            O14 = vmlsl_laneq_s16(O14, s11, c1_lo, 0); // -90
+            O15 = vmlsl_laneq_s16(O15, s11, c1_hi, 2); // -46
         }
 
-        static const int32x4_t min = vdupq_n_s32(-32768);
-        static const int32x4_t max = vdupq_n_s32(32767);
+        const int16x4_t s13 = vld1_s16(src + 13 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s13), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s13, c1_lo, 6);   //  73
+            O1 = vmlsl_laneq_s16(O1, s13, c1_hi, 4);   // -31
+            O2 = vmlsl_laneq_s16(O2, s13, c1_lo, 0);   // -90
+            O3 = vmlsl_laneq_s16(O3, s13, c1_hi, 5);   // -22
+            O4 = vmlal_laneq_s16(O4, s13, c1_lo, 5);   //  78
+            O5 = vmlal_laneq_s16(O5, s13, c1_lo, 7);   //  67
+            O6 = vmlsl_laneq_s16(O6, s13, c1_hi, 3);   // -38
+            O7 = vmlsl_laneq_s16(O7, s13, c1_lo, 1);   // -90
+            O8 = vmlsl_laneq_s16(O8, s13, c1_hi, 6);   // -13
+            O9 = vmlal_laneq_s16(O9, s13, c1_lo, 4);   //  82
+            O10 = vmlal_laneq_s16(O10, s13, c1_hi, 0); //  61
+            O11 = vmlsl_laneq_s16(O11, s13, c1_hi, 2); // -46
+            O12 = vmlsl_laneq_s16(O12, s13, c1_lo, 2); // -88
+            O13 = vmlsl_laneq_s16(O13, s13, c1_hi, 7); //  -4
+            O14 = vmlal_laneq_s16(O14, s13, c1_lo, 3); //  85
+            O15 = vmlal_laneq_s16(O15, s13, c1_hi, 1); //  54
+        }
 
+        const int16x4_t s15 = vld1_s16(src + 15 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s15), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s15, c1_lo, 7);   //  67
+            O1 = vmlsl_laneq_s16(O1, s15, c1_hi, 1);   // -54
+            O2 = vmlsl_laneq_s16(O2, s15, c1_lo, 5);   // -78
+            O3 = vmlal_laneq_s16(O3, s15, c1_hi, 3);   //  38
+            O4 = vmlal_laneq_s16(O4, s15, c1_lo, 3);   //  85
+            O5 = vmlsl_laneq_s16(O5, s15, c1_hi, 5);   // -22
+            O6 = vmlsl_laneq_s16(O6, s15, c1_lo, 1);   // -90
+            O7 = vmlal_laneq_s16(O7, s15, c1_hi, 7);   //   4
+            O8 = vmlal_laneq_s16(O8, s15, c1_lo, 0);   //  90
+            O9 = vmlal_laneq_s16(O9, s15, c1_hi, 6);   //  13
+            O10 = vmlsl_laneq_s16(O10, s15, c1_lo, 2); // -88
+            O11 = vmlsl_laneq_s16(O11, s15, c1_hi, 4); // -31
+            O12 = vmlal_laneq_s16(O12, s15, c1_lo, 4); //  82
+            O13 = vmlal_laneq_s16(O13, s15, c1_hi, 2); //  46
+            O14 = vmlsl_laneq_s16(O14, s15, c1_lo, 6); // -73
+            O15 = vmlsl_laneq_s16(O15, s15, c1_hi, 0); // -61
+        }
 
+        const int16x4_t s17 = vld1_s16(src + 17 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s17), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s17, c1_hi, 0);   //  61
+            O1 = vmlsl_laneq_s16(O1, s17, c1_lo, 6);   // -73
+            O2 = vmlsl_laneq_s16(O2, s17, c1_hi, 2);   // -46
+            O3 = vmlal_laneq_s16(O3, s17, c1_lo, 4);   //  82
+            O4 = vmlal_laneq_s16(O4, s17, c1_hi, 4);   //  31
+            O5 = vmlsl_laneq_s16(O5, s17, c1_lo, 2);   // -88
+            O6 = vmlsl_laneq_s16(O6, s17, c1_hi, 6);   // -13
+            O7 = vmlal_laneq_s16(O7, s17, c1_lo, 0);   //  90
+            O8 = vmlsl_laneq_s16(O8, s17, c1_hi, 7);   //  -4
+            O9 = vmlsl_laneq_s16(O9, s17, c1_lo, 1);   // -90
+            O10 = vmlal_laneq_s16(O10, s17, c1_hi, 5); //  22
+            O11 = vmlal_laneq_s16(O11, s17, c1_lo, 3); //  85
+            O12 = vmlsl_laneq_s16(O12, s17, c1_hi, 3); // -38
+            O13 = vmlsl_laneq_s16(O13, s17, c1_lo, 5); // -78
+            O14 = vmlal_laneq_s16(O14, s17, c1_hi, 1); //  54
+            O15 = vmlal_laneq_s16(O15, s17, c1_lo, 7); //  67
+        }
 
-X265_PRAGMA_UNROLL(16)
-        for (k = 0; k < 16; k++)
+        const int16x4_t s19 = vld1_s16(src + 19 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s19), 0) != 0)
         {
-            int32x4_t adde = vaddq_s32(vdupq_n_s32(add), Ek);
-            int32x4_t s = vaddq_s32(adde, Ok);
-            s = vshlq_s32(s, vdupq_n_s32(-shift));
-            s = vmaxq_s32(s, min);
-            s = vminq_s32(s, max);
+            O0 = vmlal_laneq_s16(O0, s19, c1_hi, 1);   //  54
+            O1 = vmlsl_laneq_s16(O1, s19, c1_lo, 3);   // -85
+            O2 = vmlsl_laneq_s16(O2, s19, c1_hi, 7);   //  -4
+            O3 = vmlal_laneq_s16(O3, s19, c1_lo, 2);   //  88
+            O4 = vmlsl_laneq_s16(O4, s19, c1_hi, 2);   // -46
+            O5 = vmlsl_laneq_s16(O5, s19, c1_hi, 0);   // -61
+            O6 = vmlal_laneq_s16(O6, s19, c1_lo, 4);   //  82
+            O7 = vmlal_laneq_s16(O7, s19, c1_hi, 6);   //  13
+            O8 = vmlsl_laneq_s16(O8, s19, c1_lo, 1);   // -90
+            O9 = vmlal_laneq_s16(O9, s19, c1_hi, 3);   //  38
+            O10 = vmlal_laneq_s16(O10, s19, c1_lo, 7); //  67
+            O11 = vmlsl_laneq_s16(O11, s19, c1_lo, 5); // -78
+            O12 = vmlsl_laneq_s16(O12, s19, c1_hi, 5); // -22
+            O13 = vmlal_laneq_s16(O13, s19, c1_lo, 0); //  90
+            O14 = vmlsl_laneq_s16(O14, s19, c1_hi, 4); // -31
+            O15 = vmlsl_laneq_s16(O15, s19, c1_lo, 6); // -73
+        }
 
+        const int16x4_t s21 = vld1_s16(src + 21 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s21), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s21, c1_hi, 2);   //  46
+            O1 = vmlsl_laneq_s16(O1, s21, c1_lo, 0);   // -90
+            O2 = vmlal_laneq_s16(O2, s21, c1_hi, 3);   //  38
+            O3 = vmlal_laneq_s16(O3, s21, c1_hi, 1);   //  54
+            O4 = vmlsl_laneq_s16(O4, s21, c1_lo, 1);   // -90
+            O5 = vmlal_laneq_s16(O5, s21, c1_hi, 4);   //  31
+            O6 = vmlal_laneq_s16(O6, s21, c1_hi, 0);   //  61
+            O7 = vmlsl_laneq_s16(O7, s21, c1_lo, 2);   // -88
+            O8 = vmlal_laneq_s16(O8, s21, c1_hi, 5);   //  22
+            O9 = vmlal_laneq_s16(O9, s21, c1_lo, 7);   //  67
+            O10 = vmlsl_laneq_s16(O10, s21, c1_lo, 3); // -85
+            O11 = vmlal_laneq_s16(O11, s21, c1_hi, 6); //  13
+            O12 = vmlal_laneq_s16(O12, s21, c1_lo, 6); //  73
+            O13 = vmlsl_laneq_s16(O13, s21, c1_lo, 4); // -82
+            O14 = vmlal_laneq_s16(O14, s21, c1_hi, 7); //   4
+            O15 = vmlal_laneq_s16(O15, s21, c1_lo, 5); //  78
+        }
 
+        const int16x4_t s23 = vld1_s16(src + 23 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s23), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s23, c1_hi, 3);   //  38
+            O1 = vmlsl_laneq_s16(O1, s23, c1_lo, 2);   // -88
+            O2 = vmlal_laneq_s16(O2, s23, c1_lo, 6);   //  73
+            O3 = vmlsl_laneq_s16(O3, s23, c1_hi, 7);   //  -4
+            O4 = vmlsl_laneq_s16(O4, s23, c1_lo, 7);   // -67
+            O5 = vmlal_laneq_s16(O5, s23, c1_lo, 1);   //  90
+            O6 = vmlsl_laneq_s16(O6, s23, c1_hi, 2);   // -46
+            O7 = vmlsl_laneq_s16(O7, s23, c1_hi, 4);   // -31
+            O8 = vmlal_laneq_s16(O8, s23, c1_lo, 3);   //  85
+            O9 = vmlsl_laneq_s16(O9, s23, c1_lo, 5);   // -78
+            O10 = vmlal_laneq_s16(O10, s23, c1_hi, 6); //  13
+            O11 = vmlal_laneq_s16(O11, s23, c1_hi, 0); //  61
+            O12 = vmlsl_laneq_s16(O12, s23, c1_lo, 0); // -90
+            O13 = vmlal_laneq_s16(O13, s23, c1_hi, 1); //  54
+            O14 = vmlal_laneq_s16(O14, s23, c1_hi, 5); //  22
+            O15 = vmlsl_laneq_s16(O15, s23, c1_lo, 4); // -82
+        }
 
-            dstk = vmovn_s32(s);
-            adde = vaddq_s32(vdupq_n_s32(add), (E15 - k));
-            s  = vsubq_s32(adde, (O15 - k));
-            s = vshlq_s32(s, vdupq_n_s32(-shift));
-            s = vmaxq_s32(s, min);
-            s = vminq_s32(s, max);
+        const int16x4_t s25 = vld1_s16(src + 25 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s25), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s25, c1_hi, 4);   //  31
+            O1 = vmlsl_laneq_s16(O1, s25, c1_lo, 5);   // -78
+            O2 = vmlal_laneq_s16(O2, s25, c1_lo, 1);   //  90
+            O3 = vmlsl_laneq_s16(O3, s25, c1_hi, 0);   // -61
+            O4 = vmlal_laneq_s16(O4, s25, c1_hi, 7);   //   4
+            O5 = vmlal_laneq_s16(O5, s25, c1_hi, 1);   //  54
+            O6 = vmlsl_laneq_s16(O6, s25, c1_lo, 2);   // -88
+            O7 = vmlal_laneq_s16(O7, s25, c1_lo, 4);   //  82
+            O8 = vmlsl_laneq_s16(O8, s25, c1_hi, 3);   // -38
+            O9 = vmlsl_laneq_s16(O9, s25, c1_hi, 5);   // -22
+            O10 = vmlal_laneq_s16(O10, s25, c1_lo, 6); //  73
+            O11 = vmlsl_laneq_s16(O11, s25, c1_lo, 0); // -90
+            O12 = vmlal_laneq_s16(O12, s25, c1_lo, 7); //  67
+            O13 = vmlsl_laneq_s16(O13, s25, c1_hi, 6); // -13
+            O14 = vmlsl_laneq_s16(O14, s25, c1_hi, 2); // -46
+            O15 = vmlal_laneq_s16(O15, s25, c1_lo, 3); //  85
+        }
 
-            dstk + 16 = vmovn_s32(s);
+        const int16x4_t s27 = vld1_s16(src + 27 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s27), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s27, c1_hi, 5);   //  22
+            O1 = vmlsl_laneq_s16(O1, s27, c1_hi, 0);   // -61
+            O2 = vmlal_laneq_s16(O2, s27, c1_lo, 3);   //  85
+            O3 = vmlsl_laneq_s16(O3, s27, c1_lo, 1);   // -90
+            O4 = vmlal_laneq_s16(O4, s27, c1_lo, 6);   //  73
+            O5 = vmlsl_laneq_s16(O5, s27, c1_hi, 3);   // -38
+            O6 = vmlsl_laneq_s16(O6, s27, c1_hi, 7);   //  -4
+            O7 = vmlal_laneq_s16(O7, s27, c1_hi, 2);   //  46
+            O8 = vmlsl_laneq_s16(O8, s27, c1_lo, 5);   // -78
+            O9 = vmlal_laneq_s16(O9, s27, c1_lo, 0);   //  90
+            O10 = vmlsl_laneq_s16(O10, s27, c1_lo, 4); // -82
+            O11 = vmlal_laneq_s16(O11, s27, c1_hi, 1); //  54
+            O12 = vmlsl_laneq_s16(O12, s27, c1_hi, 6); // -13
+            O13 = vmlsl_laneq_s16(O13, s27, c1_hi, 4); // -31
+            O14 = vmlal_laneq_s16(O14, s27, c1_lo, 7); //  67
+            O15 = vmlsl_laneq_s16(O15, s27, c1_lo, 2); // -88
         }
 
+        const int16x4_t s29 = vld1_s16(src + 29 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s29), 0) != 0)
+        {
+            O0 = vmlal_laneq_s16(O0, s29, c1_hi, 6);   //  13
+            O1 = vmlsl_laneq_s16(O1, s29, c1_hi, 3);   // -38
+            O2 = vmlal_laneq_s16(O2, s29, c1_hi, 0);   //  61
+            O3 = vmlsl_laneq_s16(O3, s29, c1_lo, 5);   // -78
+            O4 = vmlal_laneq_s16(O4, s29, c1_lo, 2);   //  88
+            O5 = vmlsl_laneq_s16(O5, s29, c1_lo, 0);   // -90
+            O6 = vmlal_laneq_s16(O6, s29, c1_lo, 3);   //  85
+            O7 = vmlsl_laneq_s16(O7, s29, c1_lo, 6);   // -73
+            O8 = vmlal_laneq_s16(O8, s29, c1_hi, 1);   //  54
+            O9 = vmlsl_laneq_s16(O9, s29, c1_hi, 4);   // -31
+            O10 = vmlal_laneq_s16(O10, s29, c1_hi, 7); //   4
+            O11 = vmlal_laneq_s16(O11, s29, c1_hi, 5); //  22
+            O12 = vmlsl_laneq_s16(O12, s29, c1_hi, 2); // -46
+            O13 = vmlal_laneq_s16(O13, s29, c1_lo, 7); //  67
+            O14 = vmlsl_laneq_s16(O14, s29, c1_lo, 4); // -82
+            O15 = vmlal_laneq_s16(O15, s29, c1_lo, 1); //  90
+        }
 
-X265_PRAGMA_UNROLL(8)
-        for (k = 0; k < 32; k += 4)
+        const int16x4_t s31 = vld1_s16(src + 31 * line + 4 * i);
+        if (vget_lane_u64(vreinterpret_u64_s16(s31), 0) != 0)
         {
-            int16x4_t x0 = dstk + 0;
-            int16x4_t x1 = dstk + 1;
-            int16x4_t x2 = dstk + 2;
-            int16x4_t x3 = dstk + 3;
-            transpose_4x4x16(x0, x1, x2, x3);
-            vst1_s16(&orig_dst0 * 32 + k, x0);
-            vst1_s16(&orig_dst1 * 32 + k, x1);
-            vst1_s16(&orig_dst2 * 32 + k, x2);
-            vst1_s16(&orig_dst3 * 32 + k, x3);
+            O0 = vmlal_laneq_s16(O0, s31, c1_hi, 7);   //   4
+            O1 = vmlsl_laneq_s16(O1, s31, c1_hi, 6);   // -13
+            O2 = vmlal_laneq_s16(O2, s31, c1_hi, 5);   //  22
+            O3 = vmlsl_laneq_s16(O3, s31, c1_hi, 4);   // -31
+            O4 = vmlal_laneq_s16(O4, s31, c1_hi, 3);   //  38
+            O5 = vmlsl_laneq_s16(O5, s31, c1_hi, 2);   // -46
+            O6 = vmlal_laneq_s16(O6, s31, c1_hi, 1);   //  54
+            O7 = vmlsl_laneq_s16(O7, s31, c1_hi, 0);   // -61
+            O8 = vmlal_laneq_s16(O8, s31, c1_lo, 7);   //  67
+            O9 = vmlsl_laneq_s16(O9, s31, c1_lo, 6);   // -73
+            O10 = vmlal_laneq_s16(O10, s31, c1_lo, 5); //  78
+            O11 = vmlsl_laneq_s16(O11, s31, c1_lo, 4); // -82
+            O12 = vmlal_laneq_s16(O12, s31, c1_lo, 3); //  85
+            O13 = vmlsl_laneq_s16(O13, s31, c1_lo, 2); // -88
+            O14 = vmlal_laneq_s16(O14, s31, c1_lo, 1); //  90
+            O15 = vmlsl_laneq_s16(O15, s31, c1_lo, 0); // -90
         }
-        orig_dst += 4 * 32;
-        src += 4;
-    }
-#undef MUL
-#undef FMA
-#undef FMAK
-#undef MULK
-#undef ODD31
-#undef ODD15
-#undef ODD7
 
-}
+        int16x4_t d_lo16;
+        int16x4_t d_hi16;
+        for (int j = 0; j < 16; j++)
+        {
+            int32x4_t t_lo = vaddq_s32(Ej, Oj);
+            d_loj = vqrshrn_n_s32(t_lo, shift);
 
+            int32x4_t t_hi = vsubq_s32(E15 - j, O15 - j);
+            d_hij = vqrshrn_n_s32(t_hi, shift);
+        }
 
+        int16x8_t d04;
+        int16x8_t d14;
+        int16x8_t d24;
+        int16x8_t d34;
+        transpose_4x8_s16(d_lo0, d_lo1, d_lo2, d_lo3, d_lo4, d_lo5, d_lo6, d_lo7,
+                          d00, d10, d20, d30);
+        transpose_4x8_s16(d_lo8, d_lo9, d_lo10, d_lo11, d_lo12, d_lo13, d_lo14, d_lo15,
+                          d01, d11, d21, d31);
+        transpose_4x8_s16(d_hi0, d_hi1, d_hi2, d_hi3, d_hi4, d_hi5, d_hi6, d_hi7,
+                          d02, d12, d22, d32);
+        transpose_4x8_s16(d_hi8, d_hi9, d_hi10, d_hi11, d_hi12, d_hi13, d_hi14, d_hi15,
+                          d03, d13, d23, d33);
+
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 0, d00);
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 1, d01);
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 2, d02);
+        vst1q_s16(dst + (4 * i + 0) * dstStride + 8 * 3, d03);
+
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 0, d10);
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 1, d11);
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 2, d12);
+        vst1q_s16(dst + (4 * i + 1) * dstStride + 8 * 3, d13);
+
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 0, d20);
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 1, d21);
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 2, d22);
+        vst1q_s16(dst + (4 * i + 2) * dstStride + 8 * 3, d23);
+
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 0, d30);
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 1, d31);
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 2, d32);
+        vst1q_s16(dst + (4 * i + 3) * dstStride + 8 * 3, d33);
+    }
 }
 
+} // namespace
+
 namespace X265_NS
 {
 // x265 private namespace
+void dst4_neon(const int16_t *src, int16_t *dst, intptr_t srcStride)
+{
+    const int shift_pass1 = 1 + X265_DEPTH - 8;
+    const int shift_pass2 = 8;
+
+    ALIGN_VAR_32(int16_t, coef4 * 4);
+    ALIGN_VAR_32(int16_t, block4 * 4);
+
+    for (int i = 0; i < 4; i++)
+    {
+        memcpy(&blocki * 4, &srci * srcStride, 4 * sizeof(int16_t));
+    }
+
+    fastForwardDst4_neon<shift_pass1>(block, coef);
+    fastForwardDst4_neon<shift_pass2>(coef, dst);
+}
+
+void dct4_neon(const int16_t *src, int16_t *dst, intptr_t srcStride)
+{
+    const int shift_pass1 = 1 + X265_DEPTH - 8;
+    const int shift_pass2 = 8;
+
+    ALIGN_VAR_32(int16_t, coef4 * 4);
+    ALIGN_VAR_32(int16_t, block4 * 4);
+
+    for (int i = 0; i < 4; i++)
+    {
+        memcpy(&blocki * 4, &srci * srcStride, 4 * sizeof(int16_t));
+    }
+
+    partialButterfly4_neon<shift_pass1>(block, coef);
+    partialButterfly4_neon<shift_pass2>(coef, dst);
+}
+
 void dct8_neon(const int16_t *src, int16_t *dst, intptr_t srcStride)
 {
     const int shift_pass1 = 2 + X265_DEPTH - 8;
@@ -1050,55 +1807,112 @@
     partialButterfly32_neon<shift_pass2>(coef, dst);
 }
 
+void idst4_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
+{
+    const int shift_pass1 = 7;
+    const int shift_pass2 = 12 - (X265_DEPTH - 8);
+
+    ALIGN_VAR_32(int16_t, coef4 * 4);
+
+    inverseDst4_neon<shift_pass1>(src, coef, 4);
+    inverseDst4_neon<shift_pass2>(coef, dst, dstStride);
+}
+
 void idct4_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
 {
-    const int shift_1st = 7;
-    const int shift_2nd = 12 - (X265_DEPTH - 8);
+    const int shift_pass1 = 7;
+    const int shift_pass2 = 12 - (X265_DEPTH - 8);
 
     ALIGN_VAR_32(int16_t, coef4 * 4);
-    ALIGN_VAR_32(int16_t, block4 * 4);
 
-    partialButterflyInverse4(src, coef, shift_1st, 4); // Forward DST BY FAST ALGORITHM, block input, coef output
-    partialButterflyInverse4(coef, block, shift_2nd, 4); // Forward DST BY FAST ALGORITHM, coef input, coeff output
+    partialButterflyInverse4_neon<shift_pass1>(src, coef, 4);
+    partialButterflyInverse4_neon<shift_pass2>(coef, dst, dstStride);
+}
 
-    for (int i = 0; i < 4; i++)
-    {
-        memcpy(&dsti * dstStride, &blocki * 4, 4 * sizeof(int16_t));
-    }
+void idct8_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
+{
+    const int shift_pass1 = 7;
+    const int shift_pass2 = 12 - (X265_DEPTH - 8);
+
+    ALIGN_VAR_32(int16_t, coef8 * 8);
+
+    partialButterflyInverse8_neon<shift_pass1>(src, coef, 8);
+    partialButterflyInverse8_neon<shift_pass2>(coef, dst, dstStride);
 }
 
 void idct16_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
 {
-    const int shift_1st = 7;
-    const int shift_2nd = 12 - (X265_DEPTH - 8);
+    const int shift_pass1 = 7;
+    const int shift_pass2 = 12 - (X265_DEPTH - 8);
 
     ALIGN_VAR_32(int16_t, coef16 * 16);
-    ALIGN_VAR_32(int16_t, block16 * 16);
 
-    partialButterflyInverse16_neon(src, coef, shift_1st, 16);
-    partialButterflyInverse16_neon(coef, block, shift_2nd, 16);
-
-    for (int i = 0; i < 16; i++)
-    {
-        memcpy(&dsti * dstStride, &blocki * 16, 16 * sizeof(int16_t));
-    }
+    partialButterflyInverse16_neon<shift_pass1>(src, coef, 16);
+    partialButterflyInverse16_neon<shift_pass2>(coef, dst, dstStride);
 }
 
 void idct32_neon(const int16_t *src, int16_t *dst, intptr_t dstStride)
 {
-    const int shift_1st = 7;
-    const int shift_2nd = 12 - (X265_DEPTH - 8);
+    const int shift_pass1 = 7;
+    const int shift_pass2 = 12 - (X265_DEPTH - 8);
 
     ALIGN_VAR_32(int16_t, coef32 * 32);
-    ALIGN_VAR_32(int16_t, block32 * 32);
 
-    partialButterflyInverse32_neon(src, coef, shift_1st, 32);
-    partialButterflyInverse32_neon(coef, block, shift_2nd, 32);
+    partialButterflyInverse32_neon<shift_pass1>(src, coef, 32);
+    partialButterflyInverse32_neon<shift_pass2>(coef, dst, dstStride);
+}
 
-    for (int i = 0; i < 32; i++)
+uint32_t findPosFirstLast_neon(const int16_t *coeff, const intptr_t trSize,
+                               const uint16_t scanTbl16)
+{
+    X265_CHECK(SCAN_SET_SIZE == 16, "SCAN_SET_SIZE must be 16\n");
+    X265_CHECK(MLS_CG_SIZE == 4, "MLS_CG_SIZE must be 4\n");
+    X265_CHECK(scanTbl2 == 1 || scanTbl2 == 2 || scanTbl2 == 8,
+               "scanTbl is invalid\n");
+
+    int16x4_t c0 = vld1_s16(&coeff0 * trSize);
+    int16x4_t c1 = vld1_s16(&coeff1 * trSize);
+    int16x4_t c2 = vld1_s16(&coeff2 * trSize);
+    int16x4_t c3 = vld1_s16(&coeff3 * trSize);
+    int16x8_t coeff01 = vcombine_s16(c0, c1);
+    int16x8_t coeff23 = vcombine_s16(c2, c3);
+
+    // Set cmp bits if coeffx != 0.
+    uint16x8_t cmp01 = vtstq_s16(coeff01, coeff01);
+    uint16x8_t cmp23 = vtstq_s16(coeff23, coeff23);
+    uint8x16_t cmp_8bit = vcombine_u8(vmovn_u16(cmp01), vmovn_u16(cmp23));
+
+    if (scanTbl2 != 2) // Skip if SCAN_HOR.
+    {
+        // Load scanTbl.
+        uint16x8_t t0 = vld1q_u16(scanTbl + 0);
+        uint16x8_t t1 = vld1q_u16(scanTbl + 8);
+        uint8x16_t scan_tbl = vcombine_u8(vmovn_u16(t0), vmovn_u16(t1));
+
+        cmp_8bit = vqtbl1q_u8(cmp_8bit, scan_tbl);
+    }
+
+    // Convert the 8x16 cmp_8bit into 4x16 cmp_4bit.
+    uint64_t cmp_4bit = vget_lane_u64(
+        vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(cmp_8bit), 4)), 0);
+
+    // NOTE: If coeff block are all zeros, the lastNZPosInCG is undefined and
+    // firstNZPosInCG is 16.
+    if (cmp_4bit == 0)
     {
-        memcpy(&dsti * dstStride, &blocki * 32, 32 * sizeof(int16_t));
+        return (uint32_t)-1 << 8 | SCAN_SET_SIZE;
     }
+
+    unsigned long id_first, id_last;
+    BSF64(id_first, cmp_4bit);
+    uint32_t firstNZPosInCG = (uint32_t)id_first >> 2;
+    BSR64(id_last, cmp_4bit);
+    uint32_t lastNZPosInCG = (uint32_t)id_last >> 2;
+
+    // Add long not needed, we only need LSB.
+    uint32_t absSumSign = (uint32_t)vaddvq_s16(vaddq_s16(coeff01, coeff23));
+
+    return (absSumSign << 31) | (lastNZPosInCG << 8) | firstNZPosInCG;
 }
 
 void setupDCTPrimitives_neon(EncoderPrimitives &p)
@@ -1111,11 +1925,15 @@
     p.cuBLOCK_8x8.psyRdoQuant = psyRdoQuant_neon<3>;
     p.cuBLOCK_16x16.psyRdoQuant = psyRdoQuant_neon<4>;
     p.cuBLOCK_32x32.psyRdoQuant = psyRdoQuant_neon<5>;
+    p.dst4x4 = dst4_neon;
+    p.cuBLOCK_4x4.dct   = dct4_neon;
     p.cuBLOCK_8x8.dct   = dct8_neon;
     p.cuBLOCK_16x16.dct = PFX(dct16_neon);
     p.cuBLOCK_32x32.dct = dct32_neon;
+    p.idst4x4 = idst4_neon;
     p.cuBLOCK_4x4.idct   = idct4_neon;
-    p.cuBLOCK_16x16.idct = PFX(idct16_neon);
+    p.cuBLOCK_8x8.idct   = idct8_neon;
+    p.cuBLOCK_16x16.idct = idct16_neon;
     p.cuBLOCK_32x32.idct = idct32_neon;
     p.cuBLOCK_4x4.count_nonzero = count_nonzero_neon<4>;
     p.cuBLOCK_8x8.count_nonzero = count_nonzero_neon<8>;
@@ -1136,11 +1954,10 @@
     p.cuBLOCK_32x32.psyRdoQuant_2p = psyRdoQuant_neon<5>;
 
     p.scanPosLast  = scanPosLast_opt;
-
+    p.findPosFirstLast = findPosFirstLast_neon;
 }
 
 };
 
 
-
 #endif

x265-4.1.tar/source/common/aarch64/dct-prim.h -> x265-4.2.tar/source/common/aarch64/dct-prim.h Changed

x265-4.1.tar/source/common/aarch64/filter-neon-dotprod.cpp -> x265-4.2.tar/source/common/aarch64/filter-neon-dotprod.cpp Changed

@@ -21,6 +21,7 @@
  * For more information, contact us at license @ x265.com.
  *****************************************************************************/
 
+#include "filter-prim.h"
 #include "filter-neon-dotprod.h"
 
 #if !HIGH_BIT_DEPTH
@@ -43,6 +44,13 @@
     3, 16, 17, 18, 7, 20, 21, 22, 11, 24, 25, 26, 15, 28, 29, 30
 };
 
+// This is to use with vtbl2q_s32_s16.
+// Extract the middle two bytes from each 32-bit element in a vector, using these byte
+// indices.
+static const uint8_t vert_shr_tbl16 = {
+    1, 2, 5, 6, 9, 10, 13, 14, 17, 18, 21, 22, 25, 26, 29, 30
+};
+
 uint8x8_t inline filter8_8_pp(uint8x16_t samples, const int8x8_t filter,
                               const int32x4_t constant, const uint8x16x3_t tbl)
 {
@@ -972,24 +980,168 @@
     }
 }
 
+template<int coeffIdx, int coeffIdy, int width, int height>
+void interp8_hv_pp_dotprod(const pixel *src, intptr_t srcStride, pixel *dst,
+                           intptr_t dstStride)
+{
+    const int v_shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    // Subtract 8 from shift since we account for that in table lookups.
+    const int v_shift_offset = v_shift - 8;
+    const int16x8_t v_filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdy);
+    const int32x4_t v_offset = vdupq_n_s32((1 << (v_shift - 1)) +
+                                           (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+
+    src -= 3 * srcStride + 3;
+
+    const uint8x16x3_t tbl = vld1q_u8_x3(dotprod_permute_tbl);
+    const int8x8_t filter = vmovn_s16(vld1q_s16(g_lumaFiltercoeffIdx));
+    const uint8x16_t shr_tbl = vld1q_u8(vert_shr_tbl);
+
+    uint8x16_t h_s11;
+    load_u8x16xn<7>(src, srcStride, h_s);
+
+    int16x4_t v_s11;
+    v_s0 = filter8_4_ps(h_s0, filter, tbl);
+    v_s1 = filter8_4_ps(h_s1, filter, tbl);
+    v_s2 = filter8_4_ps(h_s2, filter, tbl);
+    v_s3 = filter8_4_ps(h_s3, filter, tbl);
+    v_s4 = filter8_4_ps(h_s4, filter, tbl);
+    v_s5 = filter8_4_ps(h_s5, filter, tbl);
+    v_s6 = filter8_4_ps(h_s6, filter, tbl);
+
+    src += 7 * srcStride;
+
+    for (int row = 0; row < height; row += 4)
+    {
+        load_u8x16xn<4>(src, srcStride, h_s + 7);
+        v_s7 = filter8_4_ps(h_s7, filter, tbl);
+        v_s8 = filter8_4_ps(h_s8, filter, tbl);
+        v_s9 = filter8_4_ps(h_s9, filter, tbl);
+        v_s10 = filter8_4_ps(h_s10, filter, tbl);
+
+        int32x4_t sum4;
+        filter8_s16x4<coeffIdy>(v_s + 0, v_filter, v_offset, sum0);
+        filter8_s16x4<coeffIdy>(v_s + 1, v_filter, v_offset, sum1);
+        filter8_s16x4<coeffIdy>(v_s + 2, v_filter, v_offset, sum2);
+        filter8_s16x4<coeffIdy>(v_s + 3, v_filter, v_offset, sum3);
+
+        int16x8_t sum_s164;
+        sum_s160 = vtbl2q_s32_s16(sum0, sum1, shr_tbl);
+        sum_s161 = vtbl2q_s32_s16(sum2, sum3, shr_tbl);
+
+        uint8x8_t res2;
+        res0 = vqshrun_n_s16(sum_s160, v_shift_offset);
+        res1 = vqshrun_n_s16(sum_s161, v_shift_offset);
+
+        store_u8x4_strided_xN<4>(dst + 0 * dstStride, dstStride, res);
+
+        v_s0 = v_s4;
+        v_s1 = v_s5;
+        v_s2 = v_s6;
+        v_s3 = v_s7;
+        v_s4 = v_s8;
+        v_s5 = v_s9;
+        v_s6 = v_s10;
+
+        src += 4 * srcStride;
+        dst += 4 * dstStride;
+    }
+}
+
 // Declaration for use in interp_hv_pp_dotprod().
 template<int N, int width, int height>
 void interp_vert_sp_neon(const int16_t *src, intptr_t srcStride, uint8_t *dst,
                          intptr_t dstStride, int coeffIdx);
 
-// Implementation of luma_hvpp, using Neon DotProd implementation for the
-// horizontal part, and Armv8.0 Neon implementation for the vertical part.
 template<int width, int height>
 void interp_hv_pp_dotprod(const pixel *src, intptr_t srcStride, pixel *dst,
                           intptr_t dstStride, int idxX, int idxY)
 {
+// Use the merged hv paths with Clang only as performance with GCC is worse than the
+// existing approach of doing horizontal and vertical interpolation separately.
+#ifdef __clang__
+    if (width == 4 && height <= 8)
+    {
+        switch (idxX)
+        {
+        case 1:
+        {
+            switch (idxY)
+            {
+            case 1:
+                return interp8_hv_pp_dotprod<1, 1, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 2:
+                return interp8_hv_pp_dotprod<1, 2, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 3:
+                return interp8_hv_pp_dotprod<1, 3, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            }
+
+            break;
+        }
+        case 2:
+        {
+            switch (idxY)
+            {
+            case 1:
+                return interp8_hv_pp_dotprod<2, 1, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 2:
+                return interp8_hv_pp_dotprod<2, 2, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 3:
+                return interp8_hv_pp_dotprod<2, 3, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            }
+
+            break;
+        }
+        case 3:
+        {
+            switch (idxY)
+            {
+            case 1:
+                return interp8_hv_pp_dotprod<3, 1, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 2:
+                return interp8_hv_pp_dotprod<3, 2, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            case 3:
+                return interp8_hv_pp_dotprod<3, 3, width, height>(src, srcStride, dst,
+                                                                  dstStride);
+            }
+
+            break;
+        }
+        }
+    }
+    else
+    {
+        // Implementation of luma_hvpp, using Neon DotProd implementation for the
+        // horizontal part, and Armv8.0 Neon implementation for the vertical part.
+        const int N_TAPS = 8;
+
+        ALIGN_VAR_32(int16_t, immedwidth * (height + N_TAPS - 1));
+
+        interp8_horiz_ps_dotprod<width, height>(src, srcStride, immed, width, idxX,
+                                                1);
+        interp_vert_sp_neon<N_TAPS, width, height>(immed + (N_TAPS / 2 - 1) * width,
+                                                   width, dst, dstStride, idxY);
+    }
+#else // __clang__
+    // Implementation of luma_hvpp, using Neon DotProd implementation for the
+    // horizontal part, and Armv8.0 Neon implementation for the vertical part.
     const int N_TAPS = 8;
+
     ALIGN_VAR_32(int16_t, immedwidth * (height + N_TAPS - 1));
 
     interp8_horiz_ps_dotprod<width, height>(src, srcStride, immed, width, idxX,
                                             1);
     interp_vert_sp_neon<N_TAPS, width, height>(immed + (N_TAPS / 2 - 1) * width,
                                                width, dst, dstStride, idxY);
+#endif // __clang__
 }
 
 #define LUMA_DOTPROD(W, H) \

x265-4.1.tar/source/common/aarch64/filter-neon-i8mm.cpp -> x265-4.2.tar/source/common/aarch64/filter-neon-i8mm.cpp Changed

@@ -23,6 +23,7 @@
 
 #if defined(HAVE_NEON_I8MM)
 #include "filter-neon-i8mm.h"
+#include "filter-prim.h"
 #if !HIGH_BIT_DEPTH
 
 #include "mem-neon.h"
@@ -37,16 +38,17 @@
 };
 
 static const uint8_t matmul_permute_tbl232 = {
-    // Permute for luma filter 3.
+    // Permute for luma filter 1.
     { 0,  1,  2,  3,  4,  5,  6,  7,  2,  3,  4,  5,  6,  7,  8,  9,
       4,  5,  6,  7,  8,  9, 10, 11,  6,  7,  8,  9, 10, 11, 12, 13 },
-    // Permute for luma filter 1.
+    // Permute for luma filter 2 and 3.
     { 1,  2,  3,  4,  5,  6,  7,  8,  3,  4,  5,  6,  7,  8,  9, 10,
       5,  6,  7,  8,  9, 10, 11, 12,  7,  8,  9, 10, 11, 12, 13, 14 }
 };
 
-static const int8_t matmul_luma_filter216 = {
+static const int8_t matmul_luma_filter316 = {
     { -1, 4, -10, 58, 17, -5, 1, 0, 0, -1, 4, -10, 58, 17, -5, 1 },
+    { 4, -11, 40, 40, -11, 4, -1, 0, 0, 4, -11, 40, 40, -11, 4, -1 },
     { 1, -5, 17, 58, -10, 4, -1, 0, 0, 1, -5, 17, 58, -10, 4, -1 }
 };
 
@@ -59,64 +61,14 @@
     3, 16, 17, 18, 7, 20, 21, 22, 11, 24, 25, 26, 15, 28, 29, 30
 };
 
-uint8x8_t inline filter8_8_pp(uint8x16_t samples, const int8x8_t filter,
-                              const uint8x16x3_t tbl)
-{
-    // Permute input samples for dot product.
-    // { 0,  1,  2,  3,  1,  2,  3,  4,  2,  3,  4,  5,  3,  4,  5,  6 }
-    uint8x16_t perm_s0 = vqtbl1q_u8(samples, tbl.val0);
-    // { 4,  5,  6,  7,  5,  6,  7,  8,  6,  7,  8,  9,  7,  8,  9, 10 }
-    uint8x16_t perm_s1 = vqtbl1q_u8(samples, tbl.val1);
-    // { 8,  9, 10, 11,  9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 }
-    uint8x16_t perm_S2 = vqtbl1q_u8(samples, tbl.val2);
-
-    int32x4_t dotprod_lo = vusdotq_lane_s32(vdupq_n_s32(0), perm_s0, filter, 0);
-    dotprod_lo = vusdotq_lane_s32(dotprod_lo, perm_s1, filter, 1);
-    int32x4_t dotprod_hi = vusdotq_lane_s32(vdupq_n_s32(0), perm_s1, filter, 0);
-    dotprod_hi = vusdotq_lane_s32(dotprod_hi, perm_S2, filter, 1);
-
-    // Narrow and combine.
-    int16x8_t dotprod = vcombine_s16(vmovn_s32(dotprod_lo),
-                                     vmovn_s32(dotprod_hi));
-    return vqrshrun_n_s16(dotprod, IF_FILTER_PREC);
-}
-
-void inline init_sample_permute(uint8x8_t *samples, const uint8x16x3_t tbl,
-                                uint8x16_t *d)
-{
-    // Permute input samples for dot product.
-    // { 0, 1, 2, 3, 1, 2, 3, 4, 2, 3, 4, 5, 3, 4, 5, 6 }
-    d0 = vqtbl1q_u8(vcombine_u8(samples0, vdup_n_u8(0)), tbl.val0);
-    d1 = vqtbl1q_u8(vcombine_u8(samples1, vdup_n_u8(0)), tbl.val0);
-    d2 = vqtbl1q_u8(vcombine_u8(samples2, vdup_n_u8(0)), tbl.val0);
-    d3 = vqtbl1q_u8(vcombine_u8(samples3, vdup_n_u8(0)), tbl.val0);
-}
-
-uint8x8_t inline filter8_8_pp_reuse(uint8x16_t samples, const int8x8_t filter,
-                                    const uint8x16x3_t tbl, uint8x16_t &perm_s0)
-{
-    // Permute input samples for dot product.
-    // { 0,  1,  2,  3,  1,  2,  3,  4,  2,  3,  4,  5,  3,  4,  5,  6 }
-    // Already in perm_s0.
-    // { 4,  5,  6,  7,  5,  6,  7,  8,  6,  7,  8,  9,  7,  8,  9, 10 }
-    uint8x16_t perm_s1 = vqtbl1q_u8(samples, tbl.val1);
-    // { 8,  9, 10, 11,  9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 }
-    uint8x16_t perm_s2 = vqtbl1q_u8(samples, tbl.val2);
-
-    int32x4_t dotprod_lo = vusdotq_lane_s32(vdupq_n_s32(0), perm_s0, filter, 0);
-    dotprod_lo = vusdotq_lane_s32(dotprod_lo, perm_s1, filter, 1);
-    int32x4_t dotprod_hi = vusdotq_lane_s32(vdupq_n_s32(0), perm_s1, filter, 0);
-    dotprod_hi = vusdotq_lane_s32(dotprod_hi, perm_s2, filter, 1);
-
-    // Save for re-use in next iteration.
-    perm_s0 = perm_s2;
-
-    // Narrow and combine.
-    int16x8_t dotprod = vcombine_s16(vmovn_s32(dotprod_lo),
-                                     vmovn_s32(dotprod_hi));
-    return vqrshrun_n_s16(dotprod, IF_FILTER_PREC);
-}
+// This is to use with vtbl2q_s32_s16.
+// Extract the middle two bytes from each 32-bit element in a vector, using these byte
+// indices.
+static const uint8_t vert_shr_tbl16 = {
+    1, 2, 5, 6, 9, 10, 13, 14, 17, 18, 21, 22, 25, 26, 29, 30
+};
 
+template<bool coeff2>
 uint8x8_t inline filter8_8_pp_matmul(uint8x16_t samples, const int8x16_t filter,
                                      const uint8x16x2_t tbl)
 {
@@ -129,73 +81,19 @@
 
     // Narrow and combine.
     int16x8_t matmul = vcombine_s16(vmovn_s32(matmul_lo), vmovn_s32(matmul_hi));
-    return vqrshrun_n_s16(matmul, IF_FILTER_PREC);
-}
-
-int16x4_t inline filter8_4_ps(uint8x16_t samples, const int8x8_t filter,
-                              const int16x8_t constant, const uint8x16x3_t tbl)
-{
-    // Permute input samples for dot product.
-    // { 0,  1,  2,  3,  1,  2,  3,  4,  2,  3,  4,  5,  3,  4,  5,  6 }
-    uint8x16_t perm_s0 = vqtbl1q_u8(samples, tbl.val0);
-    // { 4,  5,  6,  7,  5,  6,  7,  8,  6,  7,  8,  9,  7,  8,  9, 10 }
-    uint8x16_t perm_s1 = vqtbl1q_u8(samples, tbl.val1);
-
-    int32x4_t dotprod = vusdotq_lane_s32(vdupq_n_s32(0), perm_s0, filter, 0);
-    dotprod = vusdotq_lane_s32(dotprod, perm_s1, filter, 1);
-
-    // Narrow.
-    return vadd_s16(vmovn_s32(dotprod), vget_low_s16(constant));
-}
-
-int16x8_t inline filter8_8_ps(uint8x16_t samples, const int8x8_t filter,
-                              const int16x8_t constant, const uint8x16x3_t tbl)
-{
-    // Permute input samples for dot product.
-    // { 0,  1,  2,  3,  1,  2,  3,  4,  2,  3,  4,  5,  3,  4,  5,  6 }
-    uint8x16_t perm_s0 = vqtbl1q_u8(samples, tbl.val0);
-    // { 4,  5,  6,  7,  5,  6,  7,  8,  6,  7,  8,  9,  7,  8,  9, 10 }
-    uint8x16_t perm_s1 = vqtbl1q_u8(samples, tbl.val1);
-    // { 8,  9, 10, 11,  9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 }
-    uint8x16_t perm_S2 = vqtbl1q_u8(samples, tbl.val2);
-
-    int32x4_t dotprod_lo = vusdotq_lane_s32(vdupq_n_s32(0), perm_s0, filter, 0);
-    dotprod_lo = vusdotq_lane_s32(dotprod_lo, perm_s1, filter, 1);
-    int32x4_t dotprod_hi = vusdotq_lane_s32(vdupq_n_s32(0), perm_s1, filter, 0);
-    dotprod_hi = vusdotq_lane_s32(dotprod_hi, perm_S2, filter, 1);
-
-    // Narrow and combine.
-    int16x8_t dotprod = vcombine_s16(vmovn_s32(dotprod_lo),
-                                     vmovn_s32(dotprod_hi));
-    return vaddq_s16(dotprod, constant);
-}
 
-int16x8_t inline filter8_8_ps_reuse(uint8x16_t samples, const int8x8_t filter,
-                                    const int16x8_t constant,
-                                    const uint8x16x3_t tbl, uint8x16_t &perm_s0)
-{
-    // Permute input samples for dot product.
-    // { 0,  1,  2,  3,  1,  2,  3,  4,  2,  3,  4,  5,  3,  4,  5,  6 }
-    // Already in perm_s0.
-    // { 4,  5,  6,  7,  5,  6,  7,  8,  6,  7,  8,  9,  7,  8,  9, 10 }
-    uint8x16_t perm_s1 = vqtbl1q_u8(samples, tbl.val1);
-    // { 8,  9, 10, 11,  9, 10, 11, 12, 10, 11, 12, 13, 11, 12, 13, 14 }
-    uint8x16_t perm_s2 = vqtbl1q_u8(samples, tbl.val2);
-
-    int32x4_t dotprod_lo = vusdotq_lane_s32(vdupq_n_s32(0), perm_s0, filter, 0);
-    dotprod_lo = vusdotq_lane_s32(dotprod_lo, perm_s1, filter, 1);
-    int32x4_t dotprod_hi = vusdotq_lane_s32(vdupq_n_s32(0), perm_s1, filter, 0);
-    dotprod_hi = vusdotq_lane_s32(dotprod_hi, perm_s2, filter, 1);
-
-    // Save for re-use in next iteration.
-    perm_s0 = perm_s2;
+    if (coeff2)
+    {
+        // Substract the source elements corresponding to filter tap value -1,
+        // which weren't included in the initial matrix multiplication.
+        matmul = vreinterpretq_s16_u16(vsubw_u8(vreinterpretq_u16_s16(matmul),
+                                                vget_low_u8(samples)));
+    }
 
-    // Narrow and combine.
-    int16x8_t dotprod = vcombine_s16(vmovn_s32(dotprod_lo),
-                                     vmovn_s32(dotprod_hi));
-    return vaddq_s16(dotprod, constant);
+    return vqrshrun_n_s16(matmul, IF_FILTER_PREC);
 }
 
+template<bool coeff2>
 int16x8_t inline filter8_8_ps_matmul(uint8x16_t samples, const int8x16_t filter,
                                      const int16x8_t constant,
                                      const uint8x16x2_t tbl)
@@ -209,9 +107,21 @@
 
     // Narrow and combine.
     int16x8_t matmul = vcombine_s16(vmovn_s32(matmul_lo), vmovn_s32(matmul_hi));
-    return vaddq_s16(matmul, constant);
+
+    int16x8_t offset_matmul = constant;
+
+    if (coeff2)
+    {
+        // Substract the source elements corresponding to filter tap value -1,
+        // which weren't included in the initial matrix multiplication.
+        offset_matmul = vreinterpretq_s16_u16(
+            vsubw_u8(vreinterpretq_u16_s16(offset_matmul), vget_low_u8(samples)));
+    }
+
+    return vaddq_s16(matmul, offset_matmul);
 }
 
+template<bool coeff2>
 int16x4_t inline filter8_4_ps_matmul(uint8x16_t samples, const int8x16_t filter,
                                      const int16x8_t constant,
                                      const uint8x16x2_t tbl)
@@ -221,7 +131,17 @@
 
     int32x4_t matmul = vusmmlaq_s32(vdupq_n_s32(0), perm, filter);
 
-    return vadd_s16(vmovn_s32(matmul), vget_low_s16(constant));
+    int16x8_t offset_matmul = constant;
+
+    if (coeff2)
+    {
+        // Substract the source elements corresponding to filter tap value -1,
+        // which weren't included in the initial matrix multiplication.
+        offset_matmul = vreinterpretq_s16_u16(
+            vsubw_u8(vreinterpretq_u16_s16(offset_matmul), vget_low_u8(samples)));
+    }
+
+    return vadd_s16(vmovn_s32(matmul), vget_low_s16(offset_matmul));
 }
 
 uint8x8_t inline filter4_8_pp(uint8x16_t samples, const int8x8_t filter,
@@ -334,98 +254,16 @@
 } // Unnamed namespace.
 
 namespace X265_NS {
-template<int width, int height>
-void inline interp8_horiz_pp_dotprod(const uint8_t *src, intptr_t srcStride,
-                                     uint8_t *dst, intptr_t dstStride,
-                                     int coeffIdx)
+template<bool coeff2, int width, int height>
+void inline interp8_horiz_pp_matmul(const uint8_t *src, intptr_t srcStride, uint8_t *dst,
+                                    intptr_t dstStride, int coeffIdx)
 {
     const int N_TAPS = 8;
-    src -= N_TAPS / 2 - 1;
-
-    const uint8x16x3_t tbl = vld1q_u8_x3(dotprod_permute_tbl);
-    const int8x8_t filter = vmovn_s16(vld1q_s16(g_lumaFiltercoeffIdx));
-
-    for (int row = 0; row < height; row += 4)
-    {
-        int col = 0;
-        if (width >= 32)
-        {
-            // Peel first sample permute to enable passing between iterations.
-            uint8x8_t s04;
-            load_u8x8xn<4>(src, srcStride, s0);
-            uint8x16_t ps04;
-            init_sample_permute(s0, tbl, ps0);
-
-            for (; (col + 16) <= width; col += 16)
-            {
-                uint8x16_t s_lo4, s_hi4;
-                load_u8x16xn<4>(src + col + 0, srcStride, s_lo);
-                load_u8x16xn<4>(src + col + 8, srcStride, s_hi);
-
-                uint8x8_t d_lo4;
-                d_lo0 = filter8_8_pp_reuse(s_lo0, filter, tbl, ps00);
-                d_lo1 = filter8_8_pp_reuse(s_lo1, filter, tbl, ps01);
-                d_lo2 = filter8_8_pp_reuse(s_lo2, filter, tbl, ps02);
-                d_lo3 = filter8_8_pp_reuse(s_lo3, filter, tbl, ps03);
-
-                uint8x8_t d_hi4;
-                d_hi0 = filter8_8_pp_reuse(s_hi0, filter, tbl, ps00);
-                d_hi1 = filter8_8_pp_reuse(s_hi1, filter, tbl, ps01);
-                d_hi2 = filter8_8_pp_reuse(s_hi2, filter, tbl, ps02);
-                d_hi3 = filter8_8_pp_reuse(s_hi3, filter, tbl, ps03);
-
-                store_u8x8xn<4>(dst + col + 0, dstStride, d_lo);
-                store_u8x8xn<4>(dst + col + 8, dstStride, d_hi);
-            }
-        }
-        else
-        {
-            for (; col + 8 <= width; col += 8)
-            {
-                uint8x16_t s4;
-                load_u8x16xn<4>(src + col, srcStride, s);
-
-                uint8x8_t d4;
-                d0 = filter8_8_pp(s0, filter, tbl);
-                d1 = filter8_8_pp(s1, filter, tbl);
-                d2 = filter8_8_pp(s2, filter, tbl);
-                d3 = filter8_8_pp(s3, filter, tbl);
-
-                store_u8x8xn<4>(dst + col, dstStride, d);
-            }
-        }
-        for (; col < width; col += 4)
-        {
-            uint8x16_t s4;
-            load_u8x16xn<4>(src + col, srcStride, s);
-
-            uint8x8_t d4;
-            d0 = filter8_8_pp(s0, filter, tbl);
-            d1 = filter8_8_pp(s1, filter, tbl);
-            d2 = filter8_8_pp(s2, filter, tbl);
-            d3 = filter8_8_pp(s3, filter, tbl);
-
-            store_u8x4xn<4>(dst + col, dstStride, d);
-        }
-
-        src += 4 * srcStride;
-        dst += 4 * dstStride;
-    }
-}
+    const uint8x16x2_t tbl = vld1q_u8_x2(matmul_permute_tblcoeffIdx >> 1);
+    const int8x16_t filter = vld1q_s8(matmul_luma_filtercoeffIdx - 1);
 
-template<int coeffIdx, int width, int height>
-void inline interp8_horiz_pp_matmul(const uint8_t *src, intptr_t srcStride,
-                                    uint8_t *dst, intptr_t dstStride)
-{
-    const int N_TAPS = 8;
     src -= N_TAPS / 2 - 1;
 
-    // coeffIdx is 1 or 3 for g_lumaFilter index.
-    // Select filter and permute table from the first or second array indices.
-    const int index = coeffIdx >> 1;
-    const uint8x16x2_t tbl = vld1q_u8_x2(matmul_permute_tblindex);
-    const int8x16_t filter = vld1q_s8(matmul_luma_filterindex);
-
     for (int row = 0; row < height; row += 4)
     {
         int col = 0;
@@ -438,16 +276,16 @@
                 load_u8x16xn<4>(src + col + 8, srcStride, s_hi);
 
                 uint8x8_t d_lo4;
-                d_lo0 = filter8_8_pp_matmul(s_lo0, filter, tbl);
-                d_lo1 = filter8_8_pp_matmul(s_lo1, filter, tbl);
-                d_lo2 = filter8_8_pp_matmul(s_lo2, filter, tbl);
-                d_lo3 = filter8_8_pp_matmul(s_lo3, filter, tbl);
+                d_lo0 = filter8_8_pp_matmul<coeff2>(s_lo0, filter, tbl);
+                d_lo1 = filter8_8_pp_matmul<coeff2>(s_lo1, filter, tbl);
+                d_lo2 = filter8_8_pp_matmul<coeff2>(s_lo2, filter, tbl);
+                d_lo3 = filter8_8_pp_matmul<coeff2>(s_lo3, filter, tbl);
 
                 uint8x8_t d_hi4;
-                d_hi0 = filter8_8_pp_matmul(s_hi0, filter, tbl);
-                d_hi1 = filter8_8_pp_matmul(s_hi1, filter, tbl);
-                d_hi2 = filter8_8_pp_matmul(s_hi2, filter, tbl);
-                d_hi3 = filter8_8_pp_matmul(s_hi3, filter, tbl);
+                d_hi0 = filter8_8_pp_matmul<coeff2>(s_hi0, filter, tbl);
+                d_hi1 = filter8_8_pp_matmul<coeff2>(s_hi1, filter, tbl);
+                d_hi2 = filter8_8_pp_matmul<coeff2>(s_hi2, filter, tbl);
+                d_hi3 = filter8_8_pp_matmul<coeff2>(s_hi3, filter, tbl);
 
                 store_u8x8xn<4>(dst + col + 0, dstStride, d_lo);
                 store_u8x8xn<4>(dst + col + 8, dstStride, d_hi);
@@ -461,10 +299,10 @@
                 load_u8x16xn<4>(src + col, srcStride, s);
 
                 uint8x8_t d4;
-                d0 = filter8_8_pp_matmul(s0, filter, tbl);
-                d1 = filter8_8_pp_matmul(s1, filter, tbl);
-                d2 = filter8_8_pp_matmul(s2, filter, tbl);
-                d3 = filter8_8_pp_matmul(s3, filter, tbl);
+                d0 = filter8_8_pp_matmul<coeff2>(s0, filter, tbl);
+                d1 = filter8_8_pp_matmul<coeff2>(s1, filter, tbl);
+                d2 = filter8_8_pp_matmul<coeff2>(s2, filter, tbl);
+                d3 = filter8_8_pp_matmul<coeff2>(s3, filter, tbl);
 
                 store_u8x8xn<4>(dst + col, dstStride, d);
             }
@@ -475,10 +313,10 @@
             load_u8x16xn<4>(src + col, srcStride, s);
 
             uint8x8_t d4;
-            d0 = filter8_8_pp_matmul(s0, filter, tbl);
-            d1 = filter8_8_pp_matmul(s1, filter, tbl);
-            d2 = filter8_8_pp_matmul(s2, filter, tbl);
-            d3 = filter8_8_pp_matmul(s3, filter, tbl);
+            d0 = filter8_8_pp_matmul<coeff2>(s0, filter, tbl);
+            d1 = filter8_8_pp_matmul<coeff2>(s1, filter, tbl);
+            d2 = filter8_8_pp_matmul<coeff2>(s2, filter, tbl);
+            d3 = filter8_8_pp_matmul<coeff2>(s3, filter, tbl);
 
             store_u8x4xn<4>(dst + col, dstStride, d);
         }
@@ -494,146 +332,25 @@
 {
     switch (coeffIdx)
     {
-    case 1:
-        return interp8_horiz_pp_matmul<1, width, height>(src, srcStride, dst,
-                                                         dstStride);
     case 2:
-        return interp8_horiz_pp_dotprod<width, height>(src, srcStride, dst,
-                                                       dstStride, coeffIdx);
-    case 3:
-        return interp8_horiz_pp_matmul<3, width, height>(src, srcStride, dst,
-                                                         dstStride);
+        return interp8_horiz_pp_matmul<true, width, height>(src, srcStride, dst,
+                                                            dstStride, coeffIdx);
+    default:
+        return interp8_horiz_pp_matmul<false, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx);
     }
 }
 
-template<int width, int height>
-void inline interp8_horiz_ps_dotprod(const uint8_t *src, intptr_t srcStride,
-                                     int16_t *dst, intptr_t dstStride,
-                                     int coeffIdx, int isRowExt)
-{
-    const int offset = (unsigned)-IF_INTERNAL_OFFS;
-
-    const int N_TAPS = 8;
-    int blkheight = height;
-
-    src -= N_TAPS / 2 - 1;
-    if (isRowExt)
-    {
-        src -= (N_TAPS / 2 - 1) * srcStride;
-        blkheight += N_TAPS - 1;
-    }
-
-    const uint8x16x3_t tbl = vld1q_u8_x3(dotprod_permute_tbl);
-    const int8x8_t filter = vmovn_s16(vld1q_s16(g_lumaFiltercoeffIdx));
-    const int16x8_t c = vdupq_n_s16(offset);
-
-    for (int row = 0; row + 4 <= blkheight; row += 4)
-    {
-        int col = 0;
-        if (width >= 32)
-        {
-            // Peel first sample permute to enable passing between iterations.
-            uint8x8_t s04;
-            load_u8x8xn<4>(src, srcStride, s0);
-            uint8x16_t ps04;
-            init_sample_permute(s0, tbl, ps0);
-
-            for (; col + 16 <= width; col += 16)
-            {
-                uint8x16_t s_lo4, s_hi4;
-                load_u8x16xn<4>(src + col + 0, srcStride, s_lo);
-                load_u8x16xn<4>(src + col + 8, srcStride, s_hi);
-
-                int16x8_t d_lo4;
-                d_lo0 = filter8_8_ps_reuse(s_lo0, filter, c, tbl, ps00);
-                d_lo1 = filter8_8_ps_reuse(s_lo1, filter, c, tbl, ps01);
-                d_lo2 = filter8_8_ps_reuse(s_lo2, filter, c, tbl, ps02);
-                d_lo3 = filter8_8_ps_reuse(s_lo3, filter, c, tbl, ps03);
-
-                int16x8_t d_hi4;
-                d_hi0 = filter8_8_ps_reuse(s_hi0, filter, c, tbl, ps00);
-                d_hi1 = filter8_8_ps_reuse(s_hi1, filter, c, tbl, ps01);
-                d_hi2 = filter8_8_ps_reuse(s_hi2, filter, c, tbl, ps02);
-                d_hi3 = filter8_8_ps_reuse(s_hi3, filter, c, tbl, ps03);
-
-                store_s16x8xn<4>(dst + col + 0, dstStride, d_lo);
-                store_s16x8xn<4>(dst + col + 8, dstStride, d_hi);
-            }
-        }
-        else
-        {
-            for (; col + 8 <= width; col += 8)
-            {
-                uint8x16_t s4;
-                load_u8x16xn<4>(src + col, srcStride, s);
-
-                int16x8_t d4;
-                d0 = filter8_8_ps(s0, filter, c, tbl);
-                d1 = filter8_8_ps(s1, filter, c, tbl);
-                d2 = filter8_8_ps(s2, filter, c, tbl);
-                d3 = filter8_8_ps(s3, filter, c, tbl);
-
-                store_s16x8xn<4>(dst + col, dstStride, d);
-            }
-        }
-        for (; col < width; col += 4)
-        {
-            uint8x16_t s4;
-            load_u8x16xn<4>(src + col, srcStride, s);
-
-            int16x4_t d4;
-            d0 = filter8_4_ps(s0, filter, c, tbl);
-            d1 = filter8_4_ps(s1, filter, c, tbl);
-            d2 = filter8_4_ps(s2, filter, c, tbl);
-            d3 = filter8_4_ps(s3, filter, c, tbl);
-
-            store_s16x4xn<4>(dst + col, dstStride, d);
-        }
-
-        src += 4 * srcStride;
-        dst += 4 * dstStride;
-    }
-
-    if (isRowExt)
-    {
-        // process final 3 rows
-        int col = 0;
-        for (; (col + 8) <= width; col += 8)
-        {
-            uint8x16_t s3;
-            load_u8x16xn<3>(src + col, srcStride, s);
-
-            int16x8_t d3;
-            d0 = filter8_8_ps(s0, filter, c, tbl);
-            d1 = filter8_8_ps(s1, filter, c, tbl);
-            d2 = filter8_8_ps(s2, filter, c, tbl);
-
-            store_s16x8xn<3>(dst + col, dstStride, d);
-        }
-
-        for (; col < width; col += 4)
-        {
-            uint8x16_t s3;
-            load_u8x16xn<3>(src + col, srcStride, s);
-
-            int16x4_t d3;
-            d0 = filter8_4_ps(s0, filter, c, tbl);
-            d1 = filter8_4_ps(s1, filter, c, tbl);
-            d2 = filter8_4_ps(s2, filter, c, tbl);
-
-            store_s16x4xn<3>(dst + col, dstStride, d);
-        }
-    }
-}
-
-template<int coeffIdx, int width, int height>
+template<bool coeff2, int width, int height>
 void inline interp8_horiz_ps_matmul(const uint8_t *src, intptr_t srcStride,
                                     int16_t *dst, intptr_t dstStride,
-                                    int isRowExt)
+                                    int coeffIdx, int isRowExt)
 {
     const int offset = (unsigned)-IF_INTERNAL_OFFS;
-
     const int N_TAPS = 8;
+    const uint8x16x2_t tbl = vld1q_u8_x2(matmul_permute_tblcoeffIdx >> 1);
+    const int8x16_t filter = vld1q_s8(matmul_luma_filtercoeffIdx - 1);
+    const int16x8_t c = vdupq_n_s16(offset);
     int blkheight = height;
 
     src -= N_TAPS / 2 - 1;
@@ -643,14 +360,6 @@
         blkheight += N_TAPS - 1;
     }
 
-    // coeffIdx is 1 or 3 for g_lumaFilter index.
-    // Select filter and permute table from the first or second array indices.
-    const int index = coeffIdx >> 1;
-    const uint8x16x2_t tbl = vld1q_u8_x2(matmul_permute_tblindex);
-    const int8x16_t filter = vld1q_s8(matmul_luma_filterindex);
-
-    const int16x8_t c = vdupq_n_s16(offset);
-
     for (int row = 0; row + 4 <= blkheight; row += 4)
     {
         int col = 0;
@@ -663,16 +372,16 @@
                 load_u8x16xn<4>(src + col + 8, srcStride, s_hi);
 
                 int16x8_t d_lo4;
-                d_lo0 = filter8_8_ps_matmul(s_lo0, filter, c, tbl);
-                d_lo1 = filter8_8_ps_matmul(s_lo1, filter, c, tbl);
-                d_lo2 = filter8_8_ps_matmul(s_lo2, filter, c, tbl);
-                d_lo3 = filter8_8_ps_matmul(s_lo3, filter, c, tbl);
+                d_lo0 = filter8_8_ps_matmul<coeff2>(s_lo0, filter, c, tbl);
+                d_lo1 = filter8_8_ps_matmul<coeff2>(s_lo1, filter, c, tbl);
+                d_lo2 = filter8_8_ps_matmul<coeff2>(s_lo2, filter, c, tbl);
+                d_lo3 = filter8_8_ps_matmul<coeff2>(s_lo3, filter, c, tbl);
 
                 int16x8_t d_hi4;
-                d_hi0 = filter8_8_ps_matmul(s_hi0, filter, c, tbl);
-                d_hi1 = filter8_8_ps_matmul(s_hi1, filter, c, tbl);
-                d_hi2 = filter8_8_ps_matmul(s_hi2, filter, c, tbl);
-                d_hi3 = filter8_8_ps_matmul(s_hi3, filter, c, tbl);
+                d_hi0 = filter8_8_ps_matmul<coeff2>(s_hi0, filter, c, tbl);
+                d_hi1 = filter8_8_ps_matmul<coeff2>(s_hi1, filter, c, tbl);
+                d_hi2 = filter8_8_ps_matmul<coeff2>(s_hi2, filter, c, tbl);
+                d_hi3 = filter8_8_ps_matmul<coeff2>(s_hi3, filter, c, tbl);
 
                 store_s16x8xn<4>(dst + col + 0, dstStride, d_lo);
                 store_s16x8xn<4>(dst + col + 8, dstStride, d_hi);
@@ -686,10 +395,10 @@
                 load_u8x16xn<4>(src + col, srcStride, s);
 
                 int16x8_t d4;
-                d0 = filter8_8_ps_matmul(s0, filter, c, tbl);
-                d1 = filter8_8_ps_matmul(s1, filter, c, tbl);
-                d2 = filter8_8_ps_matmul(s2, filter, c, tbl);
-                d3 = filter8_8_ps_matmul(s3, filter, c, tbl);
+                d0 = filter8_8_ps_matmul<coeff2>(s0, filter, c, tbl);
+                d1 = filter8_8_ps_matmul<coeff2>(s1, filter, c, tbl);
+                d2 = filter8_8_ps_matmul<coeff2>(s2, filter, c, tbl);
+                d3 = filter8_8_ps_matmul<coeff2>(s3, filter, c, tbl);
 
                 store_s16x8xn<4>(dst + col, dstStride, d);
             }
@@ -700,10 +409,10 @@
             load_u8x16xn<4>(src + col, srcStride, s);
 
             int16x4_t d4;
-            d0 = filter8_4_ps_matmul(s0, filter, c, tbl);
-            d1 = filter8_4_ps_matmul(s1, filter, c, tbl);
-            d2 = filter8_4_ps_matmul(s2, filter, c, tbl);
-            d3 = filter8_4_ps_matmul(s3, filter, c, tbl);
+            d0 = filter8_4_ps_matmul<coeff2>(s0, filter, c, tbl);
+            d1 = filter8_4_ps_matmul<coeff2>(s1, filter, c, tbl);
+            d2 = filter8_4_ps_matmul<coeff2>(s2, filter, c, tbl);
+            d3 = filter8_4_ps_matmul<coeff2>(s3, filter, c, tbl);
 
             store_s16x4xn<4>(dst + col, dstStride, d);
         }
@@ -722,9 +431,9 @@
             load_u8x16xn<3>(src + col, srcStride, s);
 
             int16x8_t d3;
-            d0 = filter8_8_ps_matmul(s0, filter, c, tbl);
-            d1 = filter8_8_ps_matmul(s1, filter, c, tbl);
-            d2 = filter8_8_ps_matmul(s2, filter, c, tbl);
+            d0 = filter8_8_ps_matmul<coeff2>(s0, filter, c, tbl);
+            d1 = filter8_8_ps_matmul<coeff2>(s1, filter, c, tbl);
+            d2 = filter8_8_ps_matmul<coeff2>(s2, filter, c, tbl);
 
             store_s16x8xn<3>(dst + col, dstStride, d);
         }
@@ -735,9 +444,9 @@
             load_u8x16xn<3>(src + col, srcStride, s);
 
             int16x4_t d3;
-            d0 = filter8_4_ps_matmul(s0, filter, c, tbl);
-            d1 = filter8_4_ps_matmul(s1, filter, c, tbl);
-            d2 = filter8_4_ps_matmul(s2, filter, c, tbl);
+            d0 = filter8_4_ps_matmul<coeff2>(s0, filter, c, tbl);
+            d1 = filter8_4_ps_matmul<coeff2>(s1, filter, c, tbl);
+            d2 = filter8_4_ps_matmul<coeff2>(s2, filter, c, tbl);
 
             store_s16x4xn<3>(dst + col, dstStride, d);
         }
@@ -750,16 +459,14 @@
 {
     switch (coeffIdx)
     {
-    case 1:
-        return interp8_horiz_ps_matmul<1, width, height>(src, srcStride, dst,
-                                                         dstStride, isRowExt);
     case 2:
-        return interp8_horiz_ps_dotprod<width, height>(src, srcStride, dst,
-                                                       dstStride, coeffIdx,
-                                                       isRowExt);
-    case 3:
-        return interp8_horiz_ps_matmul<3, width, height>(src, srcStride, dst,
-                                                         dstStride, isRowExt);
+        return interp8_horiz_ps_matmul<true, width, height>(src, srcStride, dst,
+                                                            dstStride, coeffIdx,
+                                                            isRowExt);
+    default:
+        return interp8_horiz_ps_matmul<false, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx,
+                                                             isRowExt);
     }
 }
 
@@ -1257,23 +964,339 @@
     }
 }
 
+template<bool coeff2, int coeffIdy, int width, int height>
+void interp8_hv_pp_i8mm(const pixel *src, intptr_t srcStride, pixel *dst,
+                        intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 8;
+    const int v_shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    // Subtract 8 from shift since we account for that in table lookups.
+    const int v_shift_offset = v_shift - 8;
+    const uint8x16x2_t tbl = vld1q_u8_x2(matmul_permute_tblcoeffIdx >> 1);
+    const int8x16_t h_filter = vld1q_s8(matmul_luma_filtercoeffIdx - 1);
+    const int16x8_t v_filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdy);
+    const int16x8_t h_offset = vdupq_n_s16((int16_t)-IF_INTERNAL_OFFS);
+    const int32x4_t v_offset = vdupq_n_s32((1 << (v_shift - 1)) +
+                                           (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+    const uint8x16_t shr_tbl = vld1q_u8(vert_shr_tbl);
+
+    src -= (N_TAPS / 2 - 1) * srcStride + (N_TAPS / 2 - 1);
+
+    int col = 0;
+    for (; col + 16 <= width; col += 16)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        uint8x16_t h_s011, h_s111;
+        int16x8_t v_s011, v_s111;
+
+        h_s00 = vld1q_u8(s + 0 * srcStride + 0);
+        v_s00 = filter8_8_ps_matmul<coeff2>(h_s00, h_filter, h_offset, tbl);
+        h_s10 = vld1q_u8(s + 0 * srcStride + 8);
+        v_s10 = filter8_8_ps_matmul<coeff2>(h_s10, h_filter, h_offset, tbl);
+
+        h_s01 = vld1q_u8(s + 1 * srcStride + 0);
+        v_s01 = filter8_8_ps_matmul<coeff2>(h_s01, h_filter, h_offset, tbl);
+        h_s11 = vld1q_u8(s + 1 * srcStride + 8);
+        v_s11 = filter8_8_ps_matmul<coeff2>(h_s11, h_filter, h_offset, tbl);
+
+        h_s02 = vld1q_u8(s + 2 * srcStride + 0);
+        v_s02 = filter8_8_ps_matmul<coeff2>(h_s02, h_filter, h_offset, tbl);
+        h_s12 = vld1q_u8(s + 2 * srcStride + 8);
+        v_s12 = filter8_8_ps_matmul<coeff2>(h_s12, h_filter, h_offset, tbl);
+
+        h_s03 = vld1q_u8(s + 3 * srcStride + 0);
+        v_s03 = filter8_8_ps_matmul<coeff2>(h_s03, h_filter, h_offset, tbl);
+        h_s13 = vld1q_u8(s + 3 * srcStride + 8);
+        v_s13 = filter8_8_ps_matmul<coeff2>(h_s13, h_filter, h_offset, tbl);
+
+        h_s04 = vld1q_u8(s + 4 * srcStride + 0);
+        v_s04 = filter8_8_ps_matmul<coeff2>(h_s04, h_filter, h_offset, tbl);
+        h_s14 = vld1q_u8(s + 4 * srcStride + 8);
+        v_s14 = filter8_8_ps_matmul<coeff2>(h_s14, h_filter, h_offset, tbl);
+
+        h_s05 = vld1q_u8(s + 5 * srcStride + 0);
+        v_s05 = filter8_8_ps_matmul<coeff2>(h_s05, h_filter, h_offset, tbl);
+        h_s15 = vld1q_u8(s + 5 * srcStride + 8);
+        v_s15 = filter8_8_ps_matmul<coeff2>(h_s15, h_filter, h_offset, tbl);
+
+        h_s06 = vld1q_u8(s + 6 * srcStride + 0);
+        v_s06 = filter8_8_ps_matmul<coeff2>(h_s06, h_filter, h_offset, tbl);
+        h_s16 = vld1q_u8(s + 6 * srcStride + 8);
+        v_s16 = filter8_8_ps_matmul<coeff2>(h_s16, h_filter, h_offset, tbl);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            uint8x8_t res_lo4, res_hi4;
+            int32x4_t sum_lo8, sum_hi8;
+
+            h_s07 = vld1q_u8(s + 0 * srcStride + 0);
+            v_s07 = filter8_8_ps_matmul<coeff2>(h_s07, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s0 + 0, v_filter, v_offset, sum_lo0, sum_hi0);
+            v_s00 = v_s04;
+            res_lo0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl),
+                                      v_shift_offset);
+
+            h_s17 = vld1q_u8(s + 0 * srcStride + 8);
+            v_s17 = filter8_8_ps_matmul<coeff2>(h_s17, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s1 + 0, v_filter, v_offset, sum_lo1, sum_hi1);
+            v_s10 = v_s14;
+            res_hi0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl),
+                                      v_shift_offset);
+
+            h_s08 = vld1q_u8(s + 1 * srcStride + 0);
+            v_s08 = filter8_8_ps_matmul<coeff2>(h_s08, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s0 + 1, v_filter, v_offset, sum_lo2, sum_hi2);
+            v_s01 = v_s05;
+            res_lo1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl),
+                                      v_shift_offset);
+
+            h_s18 = vld1q_u8(s + 1 * srcStride + 8);
+            v_s18 = filter8_8_ps_matmul<coeff2>(h_s18, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s1 + 1, v_filter, v_offset, sum_lo3, sum_hi3);
+            v_s11 = v_s15;
+            res_hi1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl),
+                                      v_shift_offset);
+
+            h_s09 = vld1q_u8(s + 2 * srcStride + 0);
+            v_s09 = filter8_8_ps_matmul<coeff2>(h_s09, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s0 + 2, v_filter, v_offset, sum_lo4, sum_hi4);
+            v_s02 = v_s06;
+            res_lo2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo4, sum_hi4, shr_tbl),
+                                      v_shift_offset);
+
+            h_s19 = vld1q_u8(s + 2 * srcStride + 8);
+            v_s19 = filter8_8_ps_matmul<coeff2>(h_s19, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s1 + 2, v_filter, v_offset, sum_lo5, sum_hi5);
+            v_s12 = v_s16;
+            res_hi2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo5, sum_hi5, shr_tbl),
+                                      v_shift_offset);
+
+            h_s010 = vld1q_u8(s + 3 * srcStride + 0);
+            v_s010 = filter8_8_ps_matmul<coeff2>(h_s010, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s0 + 3, v_filter, v_offset, sum_lo6, sum_hi6);
+            v_s03 = v_s07;
+            res_lo3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo6, sum_hi6, shr_tbl),
+                                      v_shift_offset);
+
+            h_s110 = vld1q_u8(s + 3 * srcStride + 8);
+            v_s110 = filter8_8_ps_matmul<coeff2>(h_s110, h_filter, h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s1 + 3, v_filter, v_offset, sum_lo7, sum_hi7);
+            v_s13 = v_s17;
+            res_hi3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo7, sum_hi7, shr_tbl),
+                                      v_shift_offset);
+
+            vst1q_u8(d + 0 * dstStride, vcombine_u8(res_lo0, res_hi0));
+            vst1q_u8(d + 1 * dstStride, vcombine_u8(res_lo1, res_hi1));
+            vst1q_u8(d + 2 * dstStride, vcombine_u8(res_lo2, res_hi2));
+            vst1q_u8(d + 3 * dstStride, vcombine_u8(res_lo3, res_hi3));
+
+            v_s04 = v_s08;
+            v_s14 = v_s18;
+            v_s05 = v_s09;
+            v_s15 = v_s19;
+            v_s06 = v_s010;
+            v_s16 = v_s110;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += 16;
+        dst += 16;
+    }
+
+    for (; col + 8 <= width; col += 8)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        int16x8_t v_s11;
+        v_s0 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 0 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s1 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 1 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s2 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 2 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s3 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 3 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s4 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 4 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s5 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 5 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s6 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 6 * srcStride), h_filter,
+                                             h_offset, tbl);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            uint8x8_t res4;
+            int32x4_t sum_lo4, sum_hi4;
+
+            v_s7 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 0 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s + 0, v_filter, v_offset, sum_lo0, sum_hi0);
+            v_s0 = v_s4;
+            res0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl),
+                                   v_shift_offset);
+
+            v_s8 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 1 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s + 1, v_filter, v_offset, sum_lo1, sum_hi1);
+            v_s1 = v_s5;
+            res1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl),
+                                   v_shift_offset);
+
+            v_s9 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 2 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s + 2, v_filter, v_offset, sum_lo2, sum_hi2);
+            v_s2 = v_s6;
+            res2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl),
+                                   v_shift_offset);
+
+            v_s10 = filter8_8_ps_matmul<coeff2>(vld1q_u8(s + 3 * srcStride), h_filter,
+                                                  h_offset, tbl);
+            filter8_s16x8<coeffIdy>(v_s + 3, v_filter, v_offset, sum_lo3, sum_hi3);
+            v_s3 = v_s7;
+            res3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl),
+                                   v_shift_offset);
+
+            store_u8xnxm<8, 4>(d + 0, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += 8;
+        dst += 8;
+    }
+
+    if (width % 8 != 0)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        int16x4_t v_s11;
+        v_s0 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 0 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s1 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 1 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s2 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 2 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s3 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 3 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s4 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 4 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s5 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 5 * srcStride), h_filter,
+                                             h_offset, tbl);
+        v_s6 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 6 * srcStride), h_filter,
+                                             h_offset, tbl);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            uint8x8_t res2;
+            int32x4_t sum4;
+
+            v_s7 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 0 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x4<coeffIdy>(v_s + 0, v_filter, v_offset, sum0);
+            v_s0 = v_s4;
+
+            v_s8 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 1 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x4<coeffIdy>(v_s + 1, v_filter, v_offset, sum1);
+            v_s1 = v_s5;
+
+            v_s9 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 2 * srcStride), h_filter,
+                                                 h_offset, tbl);
+            filter8_s16x4<coeffIdy>(v_s + 2, v_filter, v_offset, sum2);
+            v_s2 = v_s6;
+
+            v_s10 = filter8_4_ps_matmul<coeff2>(vld1q_u8(s + 3 * srcStride), h_filter,
+                                                  h_offset, tbl);
+            filter8_s16x4<coeffIdy>(v_s + 3, v_filter, v_offset, sum3);
+            v_s3 = v_s7;
+
+            res0 = vqshrun_n_s16(vtbl2q_s32_s16(sum0, sum1, shr_tbl),
+                                   v_shift_offset);
+            res1 = vqshrun_n_s16(vtbl2q_s32_s16(sum2, sum3, shr_tbl),
+                                   v_shift_offset);
+
+            store_u8x4_strided_xN<4>(d + 0 * dstStride, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+    }
+}
+
 // Declaration for use in interp_hv_pp_i8mm().
 template<int N, int width, int height>
 void interp_vert_sp_neon(const int16_t *src, intptr_t srcStride, uint8_t *dst,
                          intptr_t dstStride, int coeffIdx);
 
-// Implementation of luma_hvpp, using Neon i8mm implementation for the
-// horizontal part, and Armv8.0 Neon implementation for the vertical part.
 template<int width, int height>
 void interp_hv_pp_i8mm(const pixel *src, intptr_t srcStride, pixel *dst,
                        intptr_t dstStride, int idxX, int idxY)
 {
-    const int N_TAPS = 8;
-    ALIGN_VAR_32(int16_t, immedwidth * (height + N_TAPS - 1));
+// Use the merged hv paths with Clang only as performance with GCC is worse than the
+// existing approach of doing horizontal and vertical interpolation separately.
+#ifdef __clang__
+    switch (idxX)
+    {
+    case 2:
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_i8mm<true, 1, width, height>(src, srcStride, dst,
+                                                              dstStride, idxX);
+        case 2:
+            return interp8_hv_pp_i8mm<true, 2, width, height>(src, srcStride, dst,
+                                                              dstStride, idxX);
+        case 3:
+            return interp8_hv_pp_i8mm<true, 3, width, height>(src, srcStride, dst,
+                                                              dstStride, idxX);
+        }
+
+    default:
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_i8mm<false, 1, width, height>(src, srcStride, dst,
+                                                               dstStride, idxX);
+        case 2:
+            return interp8_hv_pp_i8mm<false, 2, width, height>(src, srcStride, dst,
+                                                               dstStride, idxX);
+        case 3:
+            return interp8_hv_pp_i8mm<false, 3, width, height>(src, srcStride, dst,
+                                                               dstStride, idxX);
+        }
+    }
+
+#else // __clang__
+    // Implementation of luma_hvpp, using Neon I8MM implementation for the
+    // horizontal part, and Armv8.0 Neon implementation for the vertical part.
+    const int N = 8;
+    ALIGN_VAR_32(int16_t, immedwidth * (height + N - 1));
 
     interp8_horiz_ps_i8mm<width, height>(src, srcStride, immed, width, idxX, 1);
-    interp_vert_sp_neon<N_TAPS, width, height>(immed + (N_TAPS / 2 - 1) * width,
-                                               width, dst, dstStride, idxY);
+    interp_vert_sp_neon<N, width, height>(immed + (N / 2 - 1) * width, width, dst,
+                                          dstStride, idxY);
+#endif // __clang__
 }
 
 #define LUMA_I8MM(W, H) \

x265-4.2.tar/source/common/aarch64/filter-prim-sve.cpp Added

@@ -0,0 +1,1057 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Gerda Zsejke More <gerdazsejke.more@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "filter-prim-sve.h"
+#include "mem-neon.h"
+#include "neon-sve-bridge.h"
+
+#include <arm_neon.h>
+
+#if HIGH_BIT_DEPTH
+#define SHIFT_INTERP_PS (IF_FILTER_PREC - (IF_INTERNAL_PREC - X265_DEPTH))
+
+static const uint16_t dotprod_h_permute_tbl32 = {
+    // clang-format off
+    0, 1, 2, 3, 1, 2, 3, 4,
+    2, 3, 4, 5, 3, 4, 5, 6,
+    3, 2, 1, 0, 4, 3, 2, 1,
+    5, 4, 3, 2, 6, 5, 4, 3,
+    // clang-format on
+};
+
+static const uint8_t dotprod_v_permute_tbl80 = {
+    2, 3, 4, 5, 6, 7, 16, 17, 10, 11, 12, 13, 14, 15, 18, 19,
+    2, 3, 4, 5, 6, 7, 20, 21, 10, 11, 12, 13, 14, 15, 22, 23,
+    2, 3, 4, 5, 6, 7, 24, 25, 10, 11, 12, 13, 14, 15, 26, 27,
+    2, 3, 4, 5, 6, 7, 28, 29, 10, 11, 12, 13, 14, 15, 30, 31,
+};
+
+template<bool coeff2>
+void inline filter8_u16x4(const uint16x8_t *s, uint16x4_t &d, int16x8_t filter,
+                          uint16x4_t maxVal)
+{
+    if (coeff2)
+    {
+        int16x8_t sum01 = vreinterpretq_s16_u16(vaddq_u16(s0, s1));
+        int16x8_t sum23 = vreinterpretq_s16_u16(vaddq_u16(s2, s3));
+
+        int64x2_t sum_lo = x265_sdotq_lane_s16(vdupq_n_s64(0), sum01, filter, 0);
+        int64x2_t sum_hi = x265_sdotq_lane_s16(vdupq_n_s64(0), sum23, filter, 0);
+
+        int32x4_t sum = vcombine_s32(vmovn_s64(sum_lo), vmovn_s64(sum_hi));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        int64x2_t sum_lo =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s0), filter, 0);
+        int64x2_t sum_hi =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s2), filter, 0);
+
+        sum_lo = x265_sdotq_lane_s16(sum_lo, vreinterpretq_s16_u16(s1), filter, 1);
+        sum_hi = x265_sdotq_lane_s16(sum_hi, vreinterpretq_s16_u16(s3), filter, 1);
+
+        int32x4_t sum = vcombine_s32(vmovn_s64(sum_lo), vmovn_s64(sum_hi));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<bool coeff2>
+void inline filter8_u16x8(uint16x8_t *s, uint16x8_t &d, int16x8_t filter,
+                          uint16x8_t maxVal)
+{
+    if (coeff2)
+    {
+        int16x8_t sum01 = vreinterpretq_s16_u16(vaddq_u16(s0, s1));
+        int16x8_t sum23 = vreinterpretq_s16_u16(vaddq_u16(s2, s3));
+        int16x8_t sum45 = vreinterpretq_s16_u16(vaddq_u16(s4, s5));
+        int16x8_t sum67 = vreinterpretq_s16_u16(vaddq_u16(s6, s7));
+
+        int64x2_t sum0 = x265_sdotq_lane_s16(vdupq_n_s64(0), sum01, filter, 0);
+        int64x2_t sum1 = x265_sdotq_lane_s16(vdupq_n_s64(0), sum23, filter, 0);
+        int64x2_t sum2 = x265_sdotq_lane_s16(vdupq_n_s64(0), sum45, filter, 0);
+        int64x2_t sum3 = x265_sdotq_lane_s16(vdupq_n_s64(0), sum67, filter, 0);
+
+        int32x4_t sum_lo = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum1));
+        int32x4_t sum_hi = vcombine_s32(vmovn_s64(sum2), vmovn_s64(sum3));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else
+    {
+        int64x2_t sum0 =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s0), filter, 0);
+        int64x2_t sum1 =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s1), filter, 0);
+        int64x2_t sum2 =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s2), filter, 0);
+        int64x2_t sum3 =
+            x265_sdotq_lane_s16(vdupq_n_s64(0), vreinterpretq_s16_u16(s3), filter, 0);
+
+        sum0 = x265_sdotq_lane_s16(sum0, vreinterpretq_s16_u16(s4), filter, 1);
+        sum1 = x265_sdotq_lane_s16(sum1, vreinterpretq_s16_u16(s5), filter, 1);
+        sum2 = x265_sdotq_lane_s16(sum2, vreinterpretq_s16_u16(s6), filter, 1);
+        sum3 = x265_sdotq_lane_s16(sum3, vreinterpretq_s16_u16(s7), filter, 1);
+
+        int32x4_t sum_lo = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum2));
+        int32x4_t sum_hi = vcombine_s32(vmovn_s64(sum1), vmovn_s64(sum3));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+}
+
+template<bool coeff2>
+void inline setup_s_hpp_x4(uint16x8_t *d, uint16x8_t s0, uint16x8_t s1, uint16x8_t *idx)
+{
+    if (coeff2)
+    {
+        d0 = x265_tblq_u16(s0, idx0);
+        d1 = x265_tblq_u16(s1, idx2);
+        d2 = x265_tblq_u16(s0, idx1);
+        d3 = x265_tblq_u16(s1, idx3);
+    }
+    else
+    {
+        d0 = x265_tblq_u16(s0, idx0);
+        d1 = x265_tblq_u16(s1, idx0);
+        d2 = x265_tblq_u16(s0, idx1);
+        d3 = x265_tblq_u16(s1, idx1);
+    }
+}
+
+template<bool coeff2>
+void inline setup_s_hpp_x8(uint16x8_t *d, uint16x8_t s0, uint16x8_t s1, uint16x8_t s2,
+                           uint16x8_t *idx)
+{
+    if (coeff2)
+    {
+        d0 = x265_tblq_u16(s0, idx0);
+        d1 = x265_tblq_u16(s1, idx2);
+        d2 = x265_tblq_u16(s0, idx1);
+        d3 = x265_tblq_u16(s1, idx3);
+        d4 = x265_tblq_u16(s1, idx0);
+        d5 = x265_tblq_u16(s2, idx2);
+        d6 = x265_tblq_u16(s1, idx1);
+        d7 = x265_tblq_u16(s2, idx3);
+    }
+    else
+    {
+        d0 = x265_tblq_u16(s0, idx0);
+        d1 = x265_tblq_u16(s1, idx0);
+        d2 = x265_tblq_u16(s0, idx1);
+        d3 = x265_tblq_u16(s1, idx1);
+        d4 = d1;
+        d5 = x265_tblq_u16(s2, idx0);
+        d6 = d3;
+        d7 = x265_tblq_u16(s2, idx1);
+    }
+}
+
+template<bool coeff2, int width, int height>
+void inline interp8_hpp_sve(const pixel *src, intptr_t srcStride,
+                            pixel *dst, intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 8;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    const int16x8_t filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdx);
+    uint16x8_t idx4;
+
+    idx0 = vld1q_u16(dotprod_h_permute_tbl + 0);
+    idx1 = vld1q_u16(dotprod_h_permute_tbl + 8);
+    if (coeff2)
+    {
+        idx2 = vld1q_u16(dotprod_h_permute_tbl + 16);
+        idx3 = vld1q_u16(dotprod_h_permute_tbl + 24);
+    }
+
+    src -= N_TAPS / 2 - 1;
+
+    for (int row = 0; row < height; row++)
+    {
+        if (width % 16 == 0 || width == 24)
+        {
+            int col = 0;
+            for (; col <= width - 16; col += 16)
+            {
+                uint16x8_t s5;
+                load_u16x8xn<5>(src + col, 4, s);
+
+                uint16x8_t s0N_TAPS, s1N_TAPS;
+                setup_s_hpp_x8<coeff2>(s0, s0, s1, s2, idx);
+                setup_s_hpp_x8<coeff2>(s1, s2, s3, s4, idx);
+
+                uint16x8_t d0, d1;
+                filter8_u16x8<coeff2>(s0, d0, filter, maxVal);
+                filter8_u16x8<coeff2>(s1, d1, filter, maxVal);
+
+                vst1q_u16(dst + col + 0, d0);
+                vst1q_u16(dst + col + 8, d1);
+            }
+
+            if (width == 24)
+            {
+                uint16x8_t s3;
+                load_u16x8xn<3>(src + col, 4, s);
+
+                uint16x8_t s0N_TAPS;
+                setup_s_hpp_x8<coeff2>(s0, s0, s1, s2, idx);
+
+                uint16x8_t d0;
+                filter8_u16x8<coeff2>(s0, d0, filter, maxVal);
+
+                vst1q_u16(dst + col, d0);
+            }
+        }
+        else if (width == 4)
+        {
+            uint16x8_t s2;
+            load_u16x8xn<2>(src, 4, s);
+
+            uint16x8_t s0N_TAPS;
+            setup_s_hpp_x4<coeff2>(s0, s0, s1, idx);
+
+            uint16x4_t d0;
+            filter8_u16x4<coeff2>(s0, d0, filter, vget_low_u16(maxVal));
+
+            vst1_u16(dst, d0);
+        }
+
+        src += srcStride;
+        dst += dstStride;
+    }
+}
+
+void inline filter8_ps_u16x4(const uint16x8_t *s, int16x4_t &d, int16x8_t filter,
+                             int64x2_t offset)
+{
+    int16x8_t sum01 = vreinterpretq_s16_u16(vaddq_u16(s0, s1));
+    int16x8_t sum23 = vreinterpretq_s16_u16(vaddq_u16(s2, s3));
+
+    int64x2_t sum_lo = x265_sdotq_lane_s16(offset, sum01, filter, 0);
+    int64x2_t sum_hi = x265_sdotq_lane_s16(offset, sum23, filter, 0);
+
+    int32x4_t sum = vcombine_s32(vmovn_s64(sum_lo), vmovn_s64(sum_hi));
+
+    d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+}
+
+template<int width, int height>
+void inline interp8_hps_sve(const pixel *src, intptr_t srcStride,
+                            int16_t *dst, intptr_t dstStride, int coeffIdx, int isRowExt)
+{
+    const int N_TAPS = 8;
+    int blkheight = height;
+    const int16x8_t filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdx);
+    const int64x2_t offset =
+        vdupq_n_s64((unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS);
+
+    uint16x8_t idx4;
+
+    idx0 = vld1q_u16(dotprod_h_permute_tbl + 0);
+    idx1 = vld1q_u16(dotprod_h_permute_tbl + 8);
+    idx2 = vld1q_u16(dotprod_h_permute_tbl + 16);
+    idx3 = vld1q_u16(dotprod_h_permute_tbl + 24);
+
+    if (isRowExt)
+    {
+        src -= (N_TAPS / 2 - 1) * srcStride;
+        blkheight += N_TAPS - 1;
+    }
+
+    src -= N_TAPS / 2 - 1;
+
+    for (int row = 0; row < blkheight; row++)
+    {
+        uint16x8_t s2;
+        s0 = vld1q_u16(src);
+        s1 = vld1q_u16(src + 4);
+
+        uint16x8_t s0N_TAPS;
+        setup_s_hpp_x4<true>(s0, s0, s1, idx);
+
+        int16x4_t d0;
+        filter8_ps_u16x4(s0, d0, filter, offset);
+
+        vst1_s16(dst, d0);
+
+        src += srcStride;
+        dst += dstStride;
+    }
+}
+
+void inline transpose_concat_s16_4x4(const int16x4_t s4, int16x8_t res2)
+{
+    // Transpose 16-bit elements:
+    // s0: 00, 01, 02, 03
+    // s1: 10, 11, 12, 13
+    // s2: 20, 21, 22, 23
+    // s3: 30, 31, 32, 33
+    //
+    // res0: 00 10 20 30 01 11 21 31
+    // res1: 02 12 22 32 03 13 23 33
+
+    int16x8_t s0q = vcombine_s16(s0, vdup_n_s16(0));
+    int16x8_t s1q = vcombine_s16(s1, vdup_n_s16(0));
+    int16x8_t s2q = vcombine_s16(s2, vdup_n_s16(0));
+    int16x8_t s3q = vcombine_s16(s3, vdup_n_s16(0));
+
+    int16x8_t s02 = vzip1q_s16(s0q, s2q);
+    int16x8_t s13 = vzip1q_s16(s1q, s3q);
+
+    int16x8x2_t s0123 = vzipq_s16(s02, s13);
+
+    res0 = s0123.val0;
+    res1 = s0123.val1;
+}
+
+void inline transpose_concat_s16_8x4(const int16x8_t s4, int16x8_t res4)
+{
+    // Transpose 16-bit elements:
+    // s0: 00, 01, 02, 03, 04, 05, 06, 07
+    // s1: 10, 11, 12, 13, 14, 15, 16, 17
+    // s2: 20, 21, 22, 23, 24, 25, 26, 27
+    // s3: 30, 31, 32, 33, 34, 35, 36, 37
+    //
+    // res0: 00 10 20 30 01 11 21 31
+    // res1: 02 12 22 32 03 13 23 33
+    // res2: 04 14 24 34 05 15 25 35
+    // res3: 06 16 26 36 07 17 27 37
+
+    int16x8x2_t s02 = vzipq_s16(s0, s2);
+    int16x8x2_t s13 = vzipq_s16(s1, s3);
+
+    int16x8x2_t s0123_lo = vzipq_s16(s02.val0, s13.val0);
+    int16x8x2_t s0123_hi = vzipq_s16(s02.val1, s13.val1);
+
+    res0 = s0123_lo.val0;
+    res1 = s0123_lo.val1;
+    res2 = s0123_hi.val0;
+    res3 = s0123_hi.val1;
+}
+
+void inline insert_row_into_window_s16x8(int16x8_t *s, int16x8_t s_new,
+                                         uint8x16_t *merge_block_tbl)
+{
+    int8x16x2_t samples_tbl4;
+
+    // Insert 8 new elements into a 8x4 source window represented by four uint16x8_t
+    // vectors using table lookups. Each lookup contains 16 byte indices:
+    //  - 0–15 select bytes from the original source
+    //  - 16–31 select bytes from the new row values
+
+    // { 2, 3, 4, 5, 6, 7, 16, 17, 10, 11, 12, 13, 14, 15, 18, 19 }
+    samples_tbl0.val0 = vreinterpretq_s8_s16(s0);
+    samples_tbl0.val1 = vreinterpretq_s8_s16(s_new);
+    s0 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl0, merge_block_tbl0));
+
+    // { 2, 3, 4, 5, 6, 7, 20, 21, 10, 11, 12, 13, 14, 15, 22, 23 }
+    samples_tbl1.val0 = vreinterpretq_s8_s16(s1);
+    samples_tbl1.val1 = vreinterpretq_s8_s16(s_new);
+    s1 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl1, merge_block_tbl1));
+
+    // { 2, 3, 4, 5, 6, 7, 24, 25, 10, 11, 12, 13, 14, 15, 26, 27 }
+    samples_tbl2.val0 = vreinterpretq_s8_s16(s2);
+    samples_tbl2.val1 = vreinterpretq_s8_s16(s_new);
+    s2 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl2, merge_block_tbl2));
+
+    // { 2, 3, 4, 5, 6, 7, 28, 29, 10, 11, 12, 13, 14, 15, 30, 31 }
+    samples_tbl3.val0 = vreinterpretq_s8_s16(s3);
+    samples_tbl3.val1 = vreinterpretq_s8_s16(s_new);
+    s3 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl3, merge_block_tbl3));
+}
+
+void inline insert_row_into_window_s16x4(int16x8_t *s, int16x8_t s_new,
+                                         uint8x16_t *merge_block_tbl)
+{
+    int8x16x2_t samples_tbl2;
+
+    // Insert 4 new elements into a 4x4 source window represented by two uint16x8_t
+    // vectors using table lookups. Each lookup contains 16 byte indices:
+    //  - 0–15 select bytes from the original source
+    //  - 16–31 select bytes from the new row values
+
+    // { 2, 3, 4, 5, 6, 7, 16, 17, 10, 11, 12, 13, 14, 15, 18, 19 }
+    samples_tbl0.val0 = vreinterpretq_s8_s16(s0);
+    samples_tbl0.val1 = vreinterpretq_s8_s16(s_new);
+    s0 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl0, merge_block_tbl0));
+
+    // { 2, 3, 4, 5, 6, 7, 20, 21, 10, 11, 12, 13, 14, 15, 22, 23 }
+    samples_tbl1.val0 = vreinterpretq_s8_s16(s1);
+    samples_tbl1.val1 = vreinterpretq_s8_s16(s_new);
+    s1 = vreinterpretq_s16_s8(vqtbl2q_s8(samples_tbl1, merge_block_tbl1));
+}
+
+void inline filter4_s16x4(const int16x8_t *ss, const int16x8_t filter,
+                          const int64x2_t offset, int16x4_t &d)
+{
+    int64x2_t sum0 = x265_sdotq_s16(offset, ss0, filter);
+    int64x2_t sum1 = x265_sdotq_s16(offset, ss1, filter);
+    int32x4_t sum = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum1));
+
+    d = vshrn_n_s32(sum, IF_FILTER_PREC);
+}
+
+void inline filter4_s16x8(const int16x8_t *ss, const int16x8_t filter,
+                          const int64x2_t offset, int16x8_t &d)
+{
+    int64x2_t sum0 = x265_sdotq_s16(offset, ss0, filter);
+    int64x2_t sum1 = x265_sdotq_s16(offset, ss1, filter);
+    int64x2_t sum2 = x265_sdotq_s16(offset, ss2, filter);
+    int64x2_t sum3 = x265_sdotq_s16(offset, ss3, filter);
+
+    int32x4_t sum_lo = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum1));
+    int32x4_t sum_hi = vcombine_s32(vmovn_s64(sum2), vmovn_s64(sum3));
+
+    int16x4_t d0 = vshrn_n_s32(sum_lo, IF_FILTER_PREC);
+    int16x4_t d1 = vshrn_n_s32(sum_hi, IF_FILTER_PREC);
+
+    d = vcombine_s16(d0, d1);
+}
+
+template<int width, int height>
+void inline interp4_vss_sve(const int16_t *src, intptr_t srcStride, int16_t *dst,
+                            intptr_t dstStride, const int16_t coeffIdx)
+{
+    const int N_TAPS = 4;
+    int16x4_t f = vld1_s16(X265_NS::g_chromaFiltercoeffIdx);
+    int16x8_t filter = vcombine_s16(f, f);
+    int64x2_t offset = vdupq_n_s64(0);
+    uint8x16_t merge_block_tbl4;
+
+    merge_block_tbl0 = vld1q_u8(dotprod_v_permute_tbl + 0);
+    merge_block_tbl1 = vld1q_u8(dotprod_v_permute_tbl + 16);
+    merge_block_tbl2 = vld1q_u8(dotprod_v_permute_tbl + 32);
+    merge_block_tbl3 = vld1q_u8(dotprod_v_permute_tbl + 48);
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        if (width == 12)
+        {
+            const int n_store = 8;
+            const int16_t *s = src;
+            int16_t *d = dst;
+
+            int16x8_t in4;
+            load_s16x8xn<4>(s, srcStride, in);
+            s += 4 * srcStride;
+
+            int16x8_t ss4;
+            transpose_concat_s16_8x4(in, ss);
+
+            for (int row = 0; row < height - 1; ++row)
+            {
+                int16x8_t res;
+                filter4_s16x8(ss, filter, offset, res);
+
+                store_s16xnxm<n_store, 4>(&res, d, dstStride);
+
+                int16x8_t new_r = vld1q_s16(s);
+                insert_row_into_window_s16x8(ss, new_r, merge_block_tbl);
+
+                s += srcStride;
+                d += dstStride;
+            }
+
+            int16x8_t res;
+            filter4_s16x8(ss, filter, offset, res);
+            store_s16xnxm<n_store, 4>(&res, d, dstStride);
+
+            src += 8;
+            dst += 8;
+        }
+        const int n_store = width > 4 ? 4 : width;
+
+        int16x4_t in4;
+        load_s16x4xn<4>(src, srcStride, in);
+        src += 4 * srcStride;
+
+        int16x8_t ss2;
+        transpose_concat_s16_4x4(in, ss);
+
+        for (int row = 0; row < height - 1; ++row)
+        {
+            int16x4_t res;
+            filter4_s16x4(ss, filter, offset, res);
+
+            store_s16xnxm<n_store, 1>(&res, dst, dstStride);
+
+            int16x8_t new_r = vld1q_s16(src);
+            insert_row_into_window_s16x4(ss, new_r, merge_block_tbl);
+
+            src += srcStride;
+            dst += dstStride;
+        }
+
+        int16x4_t res;
+        filter4_s16x4(ss, filter, offset, res);
+        store_s16xnxm<n_store, 1>(&res, dst, dstStride);
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 8)
+        {
+            const int16_t *s = src;
+            int16_t *d = dst;
+
+            int16x8_t in4;
+            load_s16x8xn<4>(s, srcStride, in);
+            s += 4 * srcStride;
+
+            int16x8_t ss4;
+            transpose_concat_s16_8x4(in, ss);
+
+            for (int row = 0; row < height - 1; ++row)
+            {
+                int16x8_t res;
+                filter4_s16x8(ss, filter, offset, res);
+
+                vst1q_s16(d, res);
+
+                int16x8_t new_r = vld1q_s16(s);
+                insert_row_into_window_s16x8(ss, new_r, merge_block_tbl);
+
+                s += srcStride;
+                d += dstStride;
+            }
+
+            int16x8_t res;
+            filter4_s16x8(ss, filter, offset, res);
+            vst1q_s16(d, res);
+
+            src += 8;
+            dst += 8;
+        }
+    }
+}
+
+void inline transpose_concat_u16_4x4(const uint16x4_t s4, uint16x8_t res2)
+{
+    // Transpose 16-bit elements:
+    // s0: 00, 01, 02, 03
+    // s1: 10, 11, 12, 13
+    // s2: 20, 21, 22, 23
+    // s3: 30, 31, 32, 33
+    //
+    // res0: 00 10 20 30 01 11 21 31
+    // res1: 02 12 22 32 03 13 23 33
+
+    uint16x8_t s0q = vcombine_u16(s0, vdup_n_u16(0));
+    uint16x8_t s1q = vcombine_u16(s1, vdup_n_u16(0));
+    uint16x8_t s2q = vcombine_u16(s2, vdup_n_u16(0));
+    uint16x8_t s3q = vcombine_u16(s3, vdup_n_u16(0));
+
+    uint16x8_t s02 = vzip1q_u16(s0q, s2q);
+    uint16x8_t s13 = vzip1q_u16(s1q, s3q);
+
+    uint16x8x2_t s0123 = vzipq_u16(s02, s13);
+
+    res0 = s0123.val0;
+    res1 = s0123.val1;
+}
+
+void inline transpose_concat_u16_8x4(const uint16x8_t s4, uint16x8_t res4)
+{
+    // Transpose 16-bit elements:
+    // s0: 00, 01, 02, 03, 04, 05, 06, 07
+    // s1: 10, 11, 12, 13, 14, 15, 16, 17
+    // s2: 20, 21, 22, 23, 24, 25, 26, 27
+    // s3: 30, 31, 32, 33, 34, 35, 36, 37
+    //
+    // res0: 00 10 20 30 01 11 21 31
+    // res1: 02 12 22 32 03 13 23 33
+    // res2: 04 14 24 34 05 15 25 35
+    // res3: 06 16 26 36 07 17 27 37
+
+    uint16x8x2_t s02 = vzipq_u16(s0, s2);
+    uint16x8x2_t s13 = vzipq_u16(s1, s3);
+
+    uint16x8x2_t s0123_lo = vzipq_u16(s02.val0, s13.val0);
+    uint16x8x2_t s0123_hi = vzipq_u16(s02.val1, s13.val1);
+
+    res0 = s0123_lo.val0;
+    res1 = s0123_lo.val1;
+    res2 = s0123_hi.val0;
+    res3 = s0123_hi.val1;
+}
+
+void inline insert_row_into_window_u16x8(uint16x8_t *s, uint16x8_t s_new,
+                                         uint8x16_t *merge_block_tbl)
+{
+    uint8x16x2_t samples_tbl4;
+
+    // Insert 8 new elements into a 8x4 source window represented by four uint16x8_t
+    // vectors using table lookups. Each lookup contains 16 byte indices:
+    //  - 0–15 select bytes from the original source
+    //  - 16–31 select bytes from the new row values
+
+    // { 2, 3, 4, 5, 6, 7, 16, 17, 10, 11, 12, 13, 14, 15, 18, 19 }
+    samples_tbl0.val0 = vreinterpretq_u8_u16(s0);
+    samples_tbl0.val1 = vreinterpretq_u8_u16(s_new);
+    s0 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl0, merge_block_tbl0));
+
+    // { 2, 3, 4, 5, 6, 7, 20, 21, 10, 11, 12, 13, 14, 15, 22, 23 }
+    samples_tbl1.val0 = vreinterpretq_u8_u16(s1);
+    samples_tbl1.val1 = vreinterpretq_u8_u16(s_new);
+    s1 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl1, merge_block_tbl1));
+
+    // { 2, 3, 4, 5, 6, 7, 24, 25, 10, 11, 12, 13, 14, 15, 26, 27 }
+    samples_tbl2.val0 = vreinterpretq_u8_u16(s2);
+    samples_tbl2.val1 = vreinterpretq_u8_u16(s_new);
+    s2 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl2, merge_block_tbl2));
+
+    // { 2, 3, 4, 5, 6, 7, 28, 29, 10, 11, 12, 13, 14, 15, 30, 31 }
+    samples_tbl3.val0 = vreinterpretq_u8_u16(s3);
+    samples_tbl3.val1 = vreinterpretq_u8_u16(s_new);
+    s3 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl3, merge_block_tbl3));
+}
+
+void inline insert_row_into_window_u16x4(uint16x8_t *s, uint16x8_t s_new,
+                                         uint8x16_t *merge_block_tbl)
+{
+    uint8x16x2_t samples_tbl2;
+
+    // Insert 4 new elements into a 4x4 source window represented by two uint16x8_t
+    // vectors using table lookups. Each lookup contains 16 byte indices:
+    //  - 0–15 select bytes from the original source
+    //  - 16–31 select bytes from the new row values
+
+    // { 2, 3, 4, 5, 6, 7, 16, 17, 10, 11, 12, 13, 14, 15, 18, 19 }
+    samples_tbl0.val0 = vreinterpretq_u8_u16(s0);
+    samples_tbl0.val1 = vreinterpretq_u8_u16(s_new);
+    s0 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl0, merge_block_tbl0));
+
+    // { 2, 3, 4, 5, 6, 7, 20, 21, 10, 11, 12, 13, 14, 15, 22, 23 }
+    samples_tbl1.val0 = vreinterpretq_u8_u16(s1);
+    samples_tbl1.val1 = vreinterpretq_u8_u16(s_new);
+    s1 = vreinterpretq_u16_u8(vqtbl2q_u8(samples_tbl1, merge_block_tbl1));
+}
+
+void inline filter4_u16x4(const uint16x8_t *s, const int16x8_t f2,
+                          const int64x2_t offset, const uint16x4_t maxVal,
+                          uint16x4_t &d)
+{
+    int64x2_t sum0 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s0), f2);
+    int64x2_t sum1 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s1), f2);
+
+    int32x4_t sum = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum1));
+
+    d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+    d = vmin_u16(d, maxVal);
+}
+
+void inline filter4_u16x8(const uint16x8_t *s, const int16x8_t f2,
+                          const int64x2_t offset, const uint16x8_t maxVal,
+                          uint16x8_t &d)
+{
+    int64x2_t sum0 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s0), f2);
+    int64x2_t sum1 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s1), f2);
+    int64x2_t sum2 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s2), f2);
+    int64x2_t sum3 = x265_sdotq_s16(offset, vreinterpretq_s16_u16(s3), f2);
+
+    int32x4_t sum_lo = vcombine_s32(vmovn_s64(sum0), vmovn_s64(sum1));
+    int32x4_t sum_hi = vcombine_s32(vmovn_s64(sum2), vmovn_s64(sum3));
+
+    uint16x4_t d0 = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+    uint16x4_t d1 = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+
+    d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+}
+
+template<int width, int height>
+void inline interp4_vpp_sve(const pixel *src, intptr_t srcStride, pixel *dst,
+                            intptr_t dstStride, const int16_t coeffIdx)
+{
+    const int N_TAPS = 4;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    int16x4_t f = vld1_s16(X265_NS::g_chromaFiltercoeffIdx);
+    int16x8_t filter = vcombine_s16(f, f);
+    int64x2_t offset = vdupq_n_s64(0);
+    uint8x16_t merge_block_tbl4;
+
+    merge_block_tbl0 = vld1q_u8(dotprod_v_permute_tbl + 0);
+    merge_block_tbl1 = vld1q_u8(dotprod_v_permute_tbl + 16);
+    merge_block_tbl2 = vld1q_u8(dotprod_v_permute_tbl + 32);
+    merge_block_tbl3 = vld1q_u8(dotprod_v_permute_tbl + 48);
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        if (width == 12)
+        {
+            const int n_store = 8;
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in4;
+            load_u16x8xn<4>(s, srcStride, in);
+            s += 4 * srcStride;
+
+            uint16x8_t ss4;
+            transpose_concat_u16_8x4(in, ss);
+
+            for (int row = 0; row < height - 1; ++row)
+            {
+                uint16x8_t res4;
+                filter4_u16x8(ss, filter, offset, maxVal, res0);
+
+                store_u16xnxm<n_store, 4>(d, dstStride, res);
+
+                uint16x8_t new_r = vld1q_u16(s);
+                insert_row_into_window_u16x8(ss, new_r, merge_block_tbl);
+
+                s += srcStride;
+                d += dstStride;
+            }
+
+            uint16x8_t res4;
+            filter4_u16x8(ss, filter, offset, maxVal, res0);
+            store_u16xnxm<n_store, 4>(d, dstStride, res);
+
+            src += 8;
+            dst += 8;
+        }
+        const int n_store = width > 4 ? 4 : width;
+
+        uint16x4_t in4;
+        load_u16x4xn<4>(src, srcStride, in);
+        src += 4 * srcStride;
+
+        uint16x8_t ss4;
+        transpose_concat_u16_4x4(in, ss);
+
+        for (int row = 0; row < height - 1; ++row)
+        {
+            uint16x4_t res;
+            filter4_u16x4(ss, filter, offset, vget_low_u16(maxVal), res);
+
+            store_u16xnxm<n_store, 1>(dst, dstStride, &res);
+
+            uint16x8_t new_r = vld1q_u16(src);
+            insert_row_into_window_u16x4(ss, new_r, merge_block_tbl);
+
+            src += srcStride;
+            dst += dstStride;
+        }
+
+        uint16x4_t res;
+        filter4_u16x4(ss, filter, offset, vget_low_u16(maxVal), res);
+        store_u16xnxm<n_store, 1>(dst, dstStride, &res);
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 8)
+        {
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in4;
+            load_u16x8xn<4>(s, srcStride, in);
+            s += 4 * srcStride;
+
+            uint16x8_t ss4;
+            transpose_concat_u16_8x4(in, ss);
+            for (int row = 0; row < height - 1; ++row)
+            {
+                uint16x8_t res;
+                filter4_u16x8(ss, filter, offset, maxVal, res);
+
+                vst1q_u16(d, res);
+
+                uint16x8_t new_r = vld1q_u16(s);
+                insert_row_into_window_u16x8(ss, new_r, merge_block_tbl);
+
+                s += srcStride;
+                d += dstStride;
+            }
+
+            uint16x8_t res;
+            filter4_u16x8(ss, filter, offset, maxVal, res);
+            vst1q_u16(d, res);
+
+            src += 8;
+            dst += 8;
+        }
+    }
+}
+
+namespace X265_NS {
+// Declaration for use in interp8_horiz_pp_sve().
+template<int N, int width, int height>
+void interp_horiz_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride, int coeffIdx);
+
+template<int width, int height>
+void interp8_horiz_pp_sve(const pixel *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    switch (coeffIdx)
+    {
+    case 1:
+        if (width <= 16)
+        {
+            return interp_horiz_pp_neon<8, width, height>(src, srcStride, dst,
+                                                          dstStride, coeffIdx);
+        }
+        else
+        {
+            return interp8_hpp_sve<false, width, height>(src, srcStride, dst,
+                                                         dstStride, coeffIdx);
+        }
+    case 2:
+        if (width > 4)
+        {
+            return interp_horiz_pp_neon<8, width, height>(src, srcStride, dst,
+                                                          dstStride, coeffIdx);
+        }
+        else
+        {
+            return interp8_hpp_sve<true, width, height>(src, srcStride, dst,
+                                                        dstStride, coeffIdx);
+        }
+    case 3:
+        return interp_horiz_pp_neon<8, width, height>(src, srcStride, dst,
+                                                      dstStride, coeffIdx);
+    }
+}
+
+// Declaration for use in interp8_horiz_ps_sve().
+template<int N, int width, int height>
+void interp_horiz_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride, int coeffIdx, int isRowExt);
+
+template<int width, int height>
+void interp8_horiz_ps_sve(const pixel *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride, int coeffIdx, int isRowExt)
+{
+    switch (coeffIdx)
+    {
+    case 1:
+        return interp_horiz_ps_neon<8, width, height>(src, srcStride, dst, dstStride,
+                                                      coeffIdx, isRowExt);
+    case 2:
+        return interp8_hps_sve<width, height>(src, srcStride, dst, dstStride,
+                                              coeffIdx, isRowExt);
+    case 3:
+        return interp_horiz_ps_neon<8, width, height>(src, srcStride, dst, dstStride,
+                                                      coeffIdx, isRowExt);
+    }
+}
+
+// Declaration for use in interp4_vert_ss_sve().
+template<int N, int width, int height>
+void interp_vert_ss_neon(const int16_t *src, intptr_t srcStride, int16_t *dst,
+                         intptr_t dstStride, int coeffIdx);
+
+template<int width, int height>
+void interp4_vert_ss_sve(const int16_t *src, intptr_t srcStride, int16_t *dst,
+                         intptr_t dstStride, int coeffIdx)
+{
+    switch (coeffIdx)
+    {
+    case 4:
+        return interp_vert_ss_neon<4, width, height>(src, srcStride, dst, dstStride,
+                                                     coeffIdx);
+    default:
+        return interp4_vss_sve<width, height>(src, srcStride, dst, dstStride,
+                                              coeffIdx);
+    }
+}
+
+// Declaration for use in interp4_vert_pp_sve().
+template<int N, int width, int height>
+void interp_vert_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                         intptr_t dstStride, int coeffIdx);
+
+template<int width, int height>
+void interp4_vert_pp_sve(const pixel *src, intptr_t srcStride, pixel *dst,
+                         intptr_t dstStride, int coeffIdx)
+{
+    switch (coeffIdx)
+    {
+    case 4:
+        return interp_vert_pp_neon<4, width, height>(src, srcStride, dst,
+                                                     dstStride, coeffIdx);
+    default:
+        return interp4_vpp_sve<width, height>(src, srcStride, dst,
+                                              dstStride, coeffIdx);
+    }
+}
+
+void setupFilterPrimitives_sve(EncoderPrimitives &p)
+{
+    p.puLUMA_4x4.luma_hpp    = interp8_horiz_pp_sve<4, 4>;
+    p.puLUMA_4x8.luma_hpp    = interp8_horiz_pp_sve<4, 8>;
+    p.puLUMA_4x16.luma_hpp   = interp8_horiz_pp_sve<4, 16>;
+#if X265_DEPTH == 12
+    p.puLUMA_16x4.luma_hpp   = interp8_horiz_pp_sve<16, 4>;
+    p.puLUMA_16x8.luma_hpp   = interp8_horiz_pp_sve<16, 8>;
+    p.puLUMA_16x12.luma_hpp  = interp8_horiz_pp_sve<16, 12>;
+    p.puLUMA_16x16.luma_hpp  = interp8_horiz_pp_sve<16, 16>;
+    p.puLUMA_16x32.luma_hpp  = interp8_horiz_pp_sve<16, 32>;
+    p.puLUMA_16x64.luma_hpp  = interp8_horiz_pp_sve<16, 64>;
+    p.puLUMA_24x32.luma_hpp  = interp8_horiz_pp_sve<24, 32>;
+    p.puLUMA_32x8.luma_hpp   = interp8_horiz_pp_sve<32, 8>;
+    p.puLUMA_32x16.luma_hpp  = interp8_horiz_pp_sve<32, 16>;
+    p.puLUMA_32x24.luma_hpp  = interp8_horiz_pp_sve<32, 24>;
+    p.puLUMA_32x32.luma_hpp  = interp8_horiz_pp_sve<32, 32>;
+    p.puLUMA_32x64.luma_hpp  = interp8_horiz_pp_sve<32, 64>;
+    p.puLUMA_48x64.luma_hpp  = interp8_horiz_pp_sve<48, 64>;
+    p.puLUMA_64x16.luma_hpp  = interp8_horiz_pp_sve<64, 16>;
+    p.puLUMA_64x32.luma_hpp  = interp8_horiz_pp_sve<64, 32>;
+    p.puLUMA_64x48.luma_hpp  = interp8_horiz_pp_sve<64, 48>;
+    p.puLUMA_64x64.luma_hpp  = interp8_horiz_pp_sve<64, 64>;
+#endif // X265_DEPTH == 12
+
+    p.puLUMA_4x4.luma_hps   = interp8_horiz_ps_sve<4, 4>;
+    p.puLUMA_4x8.luma_hps   = interp8_horiz_ps_sve<4, 8>;
+    p.puLUMA_4x16.luma_hps  = interp8_horiz_ps_sve<4, 16>;
+
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.filter_vss  = interp4_vert_ss_sve<8, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.filter_vss  = interp4_vert_ss_sve<8, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_12x16.filter_vss = interp4_vert_ss_sve<12, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.filter_vss  = interp4_vert_ss_sve<16, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.filter_vss  = interp4_vert_ss_sve<16, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.filter_vss = interp4_vert_ss_sve<16, 12>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.filter_vss = interp4_vert_ss_sve<16, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.filter_vss = interp4_vert_ss_sve<16, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.filter_vss = interp4_vert_ss_sve<24, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.filter_vss  = interp4_vert_ss_sve<32, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.filter_vss = interp4_vert_ss_sve<32, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.filter_vss = interp4_vert_ss_sve<32, 24>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.filter_vss = interp4_vert_ss_sve<32, 32>;
+
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.filter_vss  = interp4_vert_ss_sve<8, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x32.filter_vss  = interp4_vert_ss_sve<8, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x64.filter_vss  = interp4_vert_ss_sve<8, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.filter_vss = interp4_vert_ss_sve<12, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.filter_vss  = interp4_vert_ss_sve<16, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.filter_vss = interp4_vert_ss_sve<16, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x24.filter_vss = interp4_vert_ss_sve<16, 24>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.filter_vss = interp4_vert_ss_sve<16, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.filter_vss = interp4_vert_ss_sve<16, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_24x64.filter_vss = interp4_vert_ss_sve<24, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.filter_vss = interp4_vert_ss_sve<32, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.filter_vss = interp4_vert_ss_sve<32, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x48.filter_vss = interp4_vert_ss_sve<32, 48>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.filter_vss = interp4_vert_ss_sve<32, 64>;
+
+    p.chromaX265_CSP_I444.puLUMA_8x16.filter_vss  = interp4_vert_ss_sve<8, 16>;
+    p.chromaX265_CSP_I444.puLUMA_8x32.filter_vss  = interp4_vert_ss_sve<8, 32>;
+    p.chromaX265_CSP_I444.puLUMA_12x16.filter_vss = interp4_vert_ss_sve<12, 16>;
+    p.chromaX265_CSP_I444.puLUMA_16x4.filter_vss  = interp4_vert_ss_sve<16, 4>;
+    p.chromaX265_CSP_I444.puLUMA_16x8.filter_vss  = interp4_vert_ss_sve<16, 8>;
+    p.chromaX265_CSP_I444.puLUMA_16x12.filter_vss = interp4_vert_ss_sve<16, 12>;
+    p.chromaX265_CSP_I444.puLUMA_16x16.filter_vss = interp4_vert_ss_sve<16, 16>;
+    p.chromaX265_CSP_I444.puLUMA_16x32.filter_vss = interp4_vert_ss_sve<16, 32>;
+    p.chromaX265_CSP_I444.puLUMA_16x64.filter_vss = interp4_vert_ss_sve<16, 64>;
+    p.chromaX265_CSP_I444.puLUMA_24x32.filter_vss = interp4_vert_ss_sve<24, 32>;
+    p.chromaX265_CSP_I444.puLUMA_32x8.filter_vss  = interp4_vert_ss_sve<32, 8>;
+    p.chromaX265_CSP_I444.puLUMA_32x16.filter_vss = interp4_vert_ss_sve<32, 16>;
+    p.chromaX265_CSP_I444.puLUMA_32x24.filter_vss = interp4_vert_ss_sve<32, 24>;
+    p.chromaX265_CSP_I444.puLUMA_32x32.filter_vss = interp4_vert_ss_sve<32, 32>;
+    p.chromaX265_CSP_I444.puLUMA_32x64.filter_vss = interp4_vert_ss_sve<32, 64>;
+    p.chromaX265_CSP_I444.puLUMA_48x64.filter_vss = interp4_vert_ss_sve<48, 64>;
+    p.chromaX265_CSP_I444.puLUMA_64x16.filter_vss = interp4_vert_ss_sve<64, 16>;
+    p.chromaX265_CSP_I444.puLUMA_64x32.filter_vss = interp4_vert_ss_sve<64, 32>;
+    p.chromaX265_CSP_I444.puLUMA_64x48.filter_vss = interp4_vert_ss_sve<64, 48>;
+    p.chromaX265_CSP_I444.puLUMA_64x64.filter_vss = interp4_vert_ss_sve<64, 64>;
+
+#if X265_DEPTH == 12
+    p.chromaX265_CSP_I420.puCHROMA_420_4x4.filter_vpp   = interp4_vert_pp_sve<4, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x4.filter_vpp   = interp4_vert_pp_sve<8, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x8.filter_vpp   = interp4_vert_pp_sve<8, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.filter_vpp  = interp4_vert_pp_sve<8, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.filter_vpp  = interp4_vert_pp_sve<8, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.filter_vpp  = interp4_vert_pp_sve<16, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.filter_vpp  = interp4_vert_pp_sve<16, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.filter_vpp = interp4_vert_pp_sve<16, 12>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.filter_vpp = interp4_vert_pp_sve<16, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.filter_vpp = interp4_vert_pp_sve<16, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.filter_vpp = interp4_vert_pp_sve<24, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.filter_vpp  = interp4_vert_pp_sve<32, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.filter_vpp = interp4_vert_pp_sve<32, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.filter_vpp = interp4_vert_pp_sve<32, 24>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.filter_vpp = interp4_vert_pp_sve<32, 32>;
+
+    p.chromaX265_CSP_I422.puCHROMA_422_4x4.filter_vpp   = interp4_vert_pp_sve<4, 4>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x8.filter_vpp   = interp4_vert_pp_sve<8, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x12.filter_vpp  = interp4_vert_pp_sve<8, 12>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.filter_vpp  = interp4_vert_pp_sve<8, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x32.filter_vpp  = interp4_vert_pp_sve<8, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x64.filter_vpp  = interp4_vert_pp_sve<8, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.filter_vpp = interp4_vert_pp_sve<12, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.filter_vpp  = interp4_vert_pp_sve<16, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.filter_vpp = interp4_vert_pp_sve<16, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x24.filter_vpp = interp4_vert_pp_sve<16, 24>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.filter_vpp = interp4_vert_pp_sve<16, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.filter_vpp = interp4_vert_pp_sve<16, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_24x64.filter_vpp = interp4_vert_pp_sve<24, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.filter_vpp = interp4_vert_pp_sve<32, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.filter_vpp = interp4_vert_pp_sve<32, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x48.filter_vpp = interp4_vert_pp_sve<32, 48>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.filter_vpp = interp4_vert_pp_sve<32, 64>;
+
+    p.chromaX265_CSP_I444.puLUMA_4x4.filter_vpp   = interp4_vert_pp_sve<4, 4>;
+    p.chromaX265_CSP_I444.puLUMA_8x4.filter_vpp   = interp4_vert_pp_sve<8, 4>;
+    p.chromaX265_CSP_I444.puLUMA_8x8.filter_vpp   = interp4_vert_pp_sve<8, 8>;
+    p.chromaX265_CSP_I444.puLUMA_8x16.filter_vpp  = interp4_vert_pp_sve<8, 16>;
+    p.chromaX265_CSP_I444.puLUMA_16x4.filter_vpp  = interp4_vert_pp_sve<16, 4>;
+    p.chromaX265_CSP_I444.puLUMA_16x8.filter_vpp  = interp4_vert_pp_sve<16, 8>;
+    p.chromaX265_CSP_I444.puLUMA_16x12.filter_vpp = interp4_vert_pp_sve<16, 12>;
+    p.chromaX265_CSP_I444.puLUMA_16x16.filter_vpp = interp4_vert_pp_sve<16, 16>;
+    p.chromaX265_CSP_I444.puLUMA_16x32.filter_vpp = interp4_vert_pp_sve<16, 32>;
+    p.chromaX265_CSP_I444.puLUMA_16x64.filter_vpp = interp4_vert_pp_sve<16, 64>;
+    p.chromaX265_CSP_I444.puLUMA_24x32.filter_vpp = interp4_vert_pp_sve<24, 32>;
+    p.chromaX265_CSP_I444.puLUMA_32x8.filter_vpp  = interp4_vert_pp_sve<32, 8>;
+    p.chromaX265_CSP_I444.puLUMA_32x16.filter_vpp = interp4_vert_pp_sve<32, 16>;
+    p.chromaX265_CSP_I444.puLUMA_32x24.filter_vpp = interp4_vert_pp_sve<32, 24>;
+    p.chromaX265_CSP_I444.puLUMA_32x32.filter_vpp = interp4_vert_pp_sve<32, 32>;
+    p.chromaX265_CSP_I444.puLUMA_32x64.filter_vpp = interp4_vert_pp_sve<32, 64>;
+    p.chromaX265_CSP_I444.puLUMA_48x64.filter_vpp = interp4_vert_pp_sve<48, 64>;
+    p.chromaX265_CSP_I444.puLUMA_64x16.filter_vpp = interp4_vert_pp_sve<64, 16>;
+    p.chromaX265_CSP_I444.puLUMA_64x32.filter_vpp = interp4_vert_pp_sve<64, 32>;
+    p.chromaX265_CSP_I444.puLUMA_64x48.filter_vpp = interp4_vert_pp_sve<64, 48>;
+    p.chromaX265_CSP_I444.puLUMA_64x64.filter_vpp = interp4_vert_pp_sve<64, 64>;
+#endif // if X265_DEPTH == 12
+}
+} // namespace X265_NS
+#else // !HIGH_BIT_DEPTH
+namespace X265_NS {
+void setupFilterPrimitives_sve(EncoderPrimitives &)
+{
+}
+}
+#endif // HIGH_BIT_DEPTH

x265-4.2.tar/source/common/aarch64/filter-prim-sve.h Added

@@ -0,0 +1,37 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Gerda Zsejke More <gerdazsejke.more@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef X265_FILTER_PRIM_SVE_H
+#define X265_FILTER_PRIM_SVE_H
+
+#if defined(HAVE_SVE)
+
+#include "primitives.h"
+
+namespace X265_NS {
+void setupFilterPrimitives_sve(EncoderPrimitives &p);
+}
+
+#endif // defined(HAVE_SVE)
+
+#endif // X265_FILTER_PRIM_SVE_H

x265-4.1.tar/source/common/aarch64/filter-prim.cpp -> x265-4.2.tar/source/common/aarch64/filter-prim.cpp Changed

@@ -1,3 +1,29 @@
+/*****************************************************************************
+ * Copyright (C) 2021-2025 MulticoreWare, Inc
+ *
+ * Authors: Liwei Wang <liwei@multicorewareinc.com>
+ *          Jonathan Swinney <jswinney@amazon.com>
+ *          Hari Limaye <hari.limaye@arm.com>
+ *          Gerda Zsejke More <gerdazsejke.more@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
 #if HAVE_NEON
 
 #include "filter-prim.h"
@@ -6,235 +32,228 @@
 #include <arm_neon.h>
 
 namespace {
-void inline filter4_s16x8(int coeffIdx, const int16x8_t *s, const int16x4_t f,
-                          const int32x4_t c, int32x4_t &d0, int32x4_t &d1)
+#if !HIGH_BIT_DEPTH
+// This is to use with vtbl2q_s32_s16.
+// Extract the middle two bytes from each 32-bit element in a vector, using these byte
+// indices.
+static const uint8_t vert_shr_tbl16 = {
+    1, 2, 5, 6, 9, 10, 13, 14, 17, 18, 21, 22, 25, 26, 29, 30
+};
+#endif
+
+#if HIGH_BIT_DEPTH
+#define SHIFT_INTERP_PS (IF_FILTER_PREC - (IF_INTERNAL_PREC - X265_DEPTH))
+#endif
+
+template<bool coeff4>
+void inline filter4_s16x4_sum(const int16x4_t *s, const int16x4_t f,
+                              const int32x4_t c, int32x4_t &sum)
 {
-    if (coeffIdx == 4)
+    if (coeff4)
     {
         // { -4, 36, 36, -4 }
-        int16x8_t t0 = vaddq_s16(s1, s2);
-        int16x8_t t1 = vaddq_s16(s0, s3);
-        d0 = vmlal_n_s16(c, vget_low_s16(t0), 36);
-        d0 = vmlsl_n_s16(d0, vget_low_s16(t1), 4);
+        int16x4_t sum03 = vadd_s16(s0, s3);
+        int16x4_t sum12 = vadd_s16(s1, s2);
 
-        d1 = vmlal_n_s16(c, vget_high_s16(t0), 36);
-        d1 = vmlsl_n_s16(d1, vget_high_s16(t1), 4);
+        sum = vmlal_n_s16(c, sum12, 9);
+        sum = vsubw_s16(sum, sum03);
     }
     else
     {
-        d0 = vmlal_lane_s16(c, vget_low_s16(s0), f, 0);
-        d0 = vmlal_lane_s16(d0, vget_low_s16(s1), f, 1);
-        d0 = vmlal_lane_s16(d0, vget_low_s16(s2), f, 2);
-        d0 = vmlal_lane_s16(d0, vget_low_s16(s3), f, 3);
-
-        d1 = vmlal_lane_s16(c, vget_high_s16(s0), f, 0);
-        d1 = vmlal_lane_s16(d1, vget_high_s16(s1), f, 1);
-        d1 = vmlal_lane_s16(d1, vget_high_s16(s2), f, 2);
-        d1 = vmlal_lane_s16(d1, vget_high_s16(s3), f, 3);
+        sum = vmlal_lane_s16(c, s0, f, 0);
+        sum = vmlal_lane_s16(sum, s1, f, 1);
+        sum = vmlal_lane_s16(sum, s2, f, 2);
+        sum = vmlal_lane_s16(sum, s3, f, 3);
     }
 }
 
-template<int coeffIdx>
-void inline filter8_s16x4(const int16x4_t *s, const int32x4_t c, int32x4_t &d)
+template<bool coeff4, int shift>
+void inline filter4_ss_s16x4(const int16x4_t *s, const int16x4_t f,
+                             const int32x4_t c, int16x4_t &d)
 {
-    if (coeffIdx == 1)
-    {
-        // { -1, 4, -10, 58, 17, -5, 1, 0 }
-        d = vsubl_s16(s6, s0);
-        d = vaddq_s32(d, c);
-        d = vmlal_n_s16(d, s1, 4);
-        d = vmlsl_n_s16(d, s2, 10);
-        d = vmlal_n_s16(d, s3, 58);
-        d = vmlal_n_s16(d, s4, 17);
-        d = vmlsl_n_s16(d, s5, 5);
-    }
-    else if (coeffIdx == 2)
-    {
-        // { -1, 4, -11, 40, 40, -11, 4, -1 }
-        int32x4_t t0 = vaddl_s16(s3, s4);
-        int32x4_t t1 = vaddl_s16(s2, s5);
-        int32x4_t t2 = vaddl_s16(s1, s6);
-        int32x4_t t3 = vaddl_s16(s0, s7);
-
-        d = vmlaq_n_s32(c, t0, 40);
-        d = vmlaq_n_s32(d, t1, -11);
-        d = vmlaq_n_s32(d, t2, 4);
-        d = vmlaq_n_s32(d, t3, -1);
-    }
-    else
-    {
-        // { 0, 1, -5, 17, 58, -10, 4, -1 }
-        d = vsubl_s16(s1, s7);
-        d = vaddq_s32(d, c);
-        d = vmlal_n_s16(d, s6, 4);
-        d = vmlsl_n_s16(d, s5, 10);
-        d = vmlal_n_s16(d, s4, 58);
-        d = vmlal_n_s16(d, s3, 17);
-        d = vmlsl_n_s16(d, s2, 5);
-    }
+    int32x4_t sum;
+
+    filter4_s16x4_sum<coeff4>(s, f, c, sum);
+
+    // We divided filter values by 4 so subtract 2 from right shift in case of filter
+    // coefficient 4.
+    const int shift_offset = coeff4 ? shift - 2 : shift;
+
+    d = vshrn_n_s32(sum, shift_offset);
 }
 
-template<int coeffIdx>
-void inline filter8_s16x8(const int16x8_t *s, const int32x4_t c, int32x4_t &d0,
-                          int32x4_t &d1)
+template<bool coeff4, int shift>
+void inline filter4x2_sp_s16x4(const int16x4_t *s0, const int16x4_t *s1,
+                               const int16x4_t f, const int32x4_t c,
+                               const uint8x16_t shr_tbl, uint8x8_t &d)
 {
-    if (coeffIdx == 1)
-    {
-        // { -1, 4, -10, 58, 17, -5, 1, 0 }
-        d0 = vsubl_s16(vget_low_s16(s6), vget_low_s16(s0));
-        d0 = vaddq_s32(d0, c);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s1), 4);
-        d0 = vmlsl_n_s16(d0, vget_low_s16(s2), 10);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s3), 58);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s4), 17);
-        d0 = vmlsl_n_s16(d0, vget_low_s16(s5), 5);
-
-        d1 = vsubl_s16(vget_high_s16(s6), vget_high_s16(s0));
-        d1 = vaddq_s32(d1, c);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s1), 4);
-        d1 = vmlsl_n_s16(d1, vget_high_s16(s2), 10);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s3), 58);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s4), 17);
-        d1 = vmlsl_n_s16(d1, vget_high_s16(s5), 5);
-    }
-    else if (coeffIdx == 2)
+    int32x4_t sum0, sum1;
+
+    filter4_s16x4_sum<coeff4>(s0, f, c, sum0);
+    filter4_s16x4_sum<coeff4>(s1, f, c, sum1);
+    int16x8_t sum = vtbl2q_s32_s16(sum0, sum1, shr_tbl);
+
+    // We divided filter values by 4 so subtract 2 from right shift in case of filter
+    // coefficient 4.
+    const int shift_offset = coeff4 ? shift - 2 : shift;
+
+    d = vqshrun_n_s16(sum, shift_offset);
+}
+
+template<bool coeff4>
+void inline filter4_s16x8_sum(const int16x8_t *s, const int16x4_t f,
+                              const int32x4_t c, int32x4_t &sum_lo, int32x4_t &sum_hi)
+{
+    if (coeff4)
     {
-        // { -1, 4, -11, 40, 40, -11, 4, -1 }
-        int32x4_t t0 = vaddl_s16(vget_low_s16(s3), vget_low_s16(s4));
-        int32x4_t t1 = vaddl_s16(vget_low_s16(s2), vget_low_s16(s5));
-        int32x4_t t2 = vaddl_s16(vget_low_s16(s1), vget_low_s16(s6));
-        int32x4_t t3 = vaddl_s16(vget_low_s16(s0), vget_low_s16(s7));
-
-        d0 = vmlaq_n_s32(c, t0, 40);
-        d0 = vmlaq_n_s32(d0, t1, -11);
-        d0 = vmlaq_n_s32(d0, t2, 4);
-        d0 = vmlaq_n_s32(d0, t3, -1);
-
-        int32x4_t t4 = vaddl_s16(vget_high_s16(s3), vget_high_s16(s4));
-        int32x4_t t5 = vaddl_s16(vget_high_s16(s2), vget_high_s16(s5));
-        int32x4_t t6 = vaddl_s16(vget_high_s16(s1), vget_high_s16(s6));
-        int32x4_t t7 = vaddl_s16(vget_high_s16(s0), vget_high_s16(s7));
-
-        d1 = vmlaq_n_s32(c, t4, 40);
-        d1 = vmlaq_n_s32(d1, t5, -11);
-        d1 = vmlaq_n_s32(d1, t6, 4);
-        d1 = vmlaq_n_s32(d1, t7, -1);
+        // { -4, 36, 36, -4 }
+        int16x8_t sum03 = vaddq_s16(s0, s3);
+        int16x8_t sum12 = vaddq_s16(s1, s2);
+
+        sum_lo = vmlal_n_s16(c, vget_low_s16(sum12), 9);
+        sum_hi = vmlal_n_s16(c, vget_high_s16(sum12), 9);
+
+        sum_lo = vsubw_s16(sum_lo, vget_low_s16(sum03));
+        sum_hi = vsubw_s16(sum_hi, vget_high_s16(sum03));
     }
     else
     {
-        // { 0, 1, -5, 17, 58, -10, 4, -1 }
-        d0 = vsubl_s16(vget_low_s16(s1), vget_low_s16(s7));
-        d0 = vaddq_s32(d0, c);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s6), 4);
-        d0 = vmlsl_n_s16(d0, vget_low_s16(s5), 10);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s4), 58);
-        d0 = vmlal_n_s16(d0, vget_low_s16(s3), 17);
-        d0 = vmlsl_n_s16(d0, vget_low_s16(s2), 5);
-
-        d1 = vsubl_s16(vget_high_s16(s1), vget_high_s16(s7));
-        d1 = vaddq_s32(d1, c);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s6), 4);
-        d1 = vmlsl_n_s16(d1, vget_high_s16(s5), 10);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s4), 58);
-        d1 = vmlal_n_s16(d1, vget_high_s16(s3), 17);
-        d1 = vmlsl_n_s16(d1, vget_high_s16(s2), 5);
+        sum_lo = vmlal_lane_s16(c, vget_low_s16(s0), f, 0);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s1), f, 1);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s2), f, 2);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s3), f, 3);
+
+        sum_hi = vmlal_lane_s16(c, vget_high_s16(s0), f, 0);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s1), f, 1);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s2), f, 2);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s3), f, 3);
     }
 }
 
-template<int width, int height>
+template<bool coeff4, int shift>
+void inline filter4_ss_s16x8(const int16x8_t *s, const int16x4_t f,
+                             const int32x4_t c, int16x8_t &d)
+{
+    int32x4_t sum_lo, sum_hi;
+
+    filter4_s16x8_sum<coeff4>(s, f, c, sum_lo, sum_hi);
+
+    // We divided filter values by 4 so subtract 2 from right shift in case of filter
+    // coefficient 4.
+    const int shift_offset = coeff4 ? shift - 2 : shift;
+
+    d = vcombine_s16(vshrn_n_s32(sum_lo, shift_offset),
+                     vshrn_n_s32(sum_hi, shift_offset));
+}
+
+template<bool coeff4, int shift>
+void inline filter4_sp_s16x8(const int16x8_t *s, const int16x4_t f,
+                             const int32x4_t c, const uint8x16_t shr_tbl, uint8x8_t &d)
+{
+    int32x4_t sum_lo, sum_hi;
+
+    filter4_s16x8_sum<coeff4>(s, f, c, sum_lo, sum_hi);
+
+    int16x8_t sum = vtbl2q_s32_s16(sum_lo, sum_hi, shr_tbl);
+
+    // We divided filter values by 4 so subtract 2 from right shift in case of filter
+    // coefficient 4.
+    const int shift_offset = coeff4 ? shift - 2 : shift;
+
+    d = vqshrun_n_s16(sum, shift_offset);
+}
+
+template<bool coeff4, int width, int height>
 void interp4_vert_ss_neon(const int16_t *src, intptr_t srcStride, int16_t *dst,
                           intptr_t dstStride, int coeffIdx)
 {
     const int N_TAPS = 4;
-    src -= (N_TAPS / 2 - 1) * srcStride;
-
+    const int shift = IF_FILTER_PREC;
     const int16x4_t filter = vld1_s16(X265_NS::g_chromaFiltercoeffIdx);
-
     // Zero constant in order to use filter helper functions (optimised away).
     const int32x4_t c = vdupq_n_s32(0);
 
-    if (width == 12)
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
     {
-        const int16_t *s = src;
-        int16_t *d = dst;
+        if (width == 12 || width == 6)
+        {
+            const int n_store = width == 12 ? 8 : 6;
+            const int16_t *s = src;
+            int16_t *d = dst;
 
-        int16x8_t in7;
-        load_s16x8xn<3>(s, srcStride, in);
-        s += 3 * srcStride;
+            int16x8_t in7;
+            load_s16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
 
-        for (int row = 0; (row + 4) <= height; row += 4)
-        {
-            load_s16x8xn<4>(s, srcStride, in + 3);
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 3);
 
-            int32x4_t sum_lo4;
-            int32x4_t sum_hi4;
-            filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0, sum_hi0);
-            filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1, sum_hi1);
-            filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2, sum_hi2);
-            filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3, sum_hi3);
+                int16x8_t res4;
+                filter4_ss_s16x8<coeff4, shift>(in + 0, filter, c, res0);
+                filter4_ss_s16x8<coeff4, shift>(in + 1, filter, c, res1);
+                filter4_ss_s16x8<coeff4, shift>(in + 2, filter, c, res2);
+                filter4_ss_s16x8<coeff4, shift>(in + 3, filter, c, res3);
 
-            int16x8_t sum4;
-            sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi0, IF_FILTER_PREC));
-            sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi1, IF_FILTER_PREC));
-            sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi2, IF_FILTER_PREC));
-            sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi3, IF_FILTER_PREC));
+                store_s16xnxm<n_store, 4>(res, d, dstStride);
 
-            store_s16x8xn<4>(d, dstStride, sum);
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
 
-            in0 = in4;
-            in1 = in5;
-            in2 = in6;
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
 
-            s += 4 * srcStride;
-            d += 4 * dstStride;
-        }
+            if (width == 6)
+            {
+                return;
+            }
 
-        src += 8;
-        dst += 8;
-        s = src;
-        d = dst;
+            src += 8;
+            dst += 8;
+        }
 
-        load_s16x8xn<3>(s, srcStride, in);
-        s += 3 * srcStride;
+        int16x4_t in7;
+        load_s16x4xn<3>(src, srcStride, in);
+        src += 3 * srcStride;
 
-        for (int row = 0; (row + 4) <= height; row += 4)
+        const int n_store = width > 4 ? 4 : width;
+        for (int row = 0; row + 4 <= height; row += 4)
         {
-            load_s16x8xn<4>(s, srcStride, in + 3);
-
-            int32x4_t sum_lo4;
-            int32x4_t sum_hi4;
-            filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0, sum_hi0);
-            filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1, sum_hi1);
-            filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2, sum_hi2);
-            filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3, sum_hi3);
+            load_s16x4xn<4>(src, srcStride, in + 3);
 
-            int16x8_t sum4;
-            sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi0, IF_FILTER_PREC));
-            sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi1, IF_FILTER_PREC));
-            sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi2, IF_FILTER_PREC));
-            sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, IF_FILTER_PREC),
-                                  vshrn_n_s32(sum_hi3, IF_FILTER_PREC));
+            int16x4_t res4;
+            filter4_ss_s16x4<coeff4, shift>(in + 0, filter, c, res0);
+            filter4_ss_s16x4<coeff4, shift>(in + 1, filter, c, res1);
+            filter4_ss_s16x4<coeff4, shift>(in + 2, filter, c, res2);
+            filter4_ss_s16x4<coeff4, shift>(in + 3, filter, c, res3);
 
-            store_s16x4xn<4>(d, dstStride, sum);
+            store_s16xnxm<n_store, 4>(res, dst, dstStride);
 
             in0 = in4;
             in1 = in5;
             in2 = in6;
 
-            s += 4 * srcStride;
-            d += 4 * dstStride;
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_s16x4xn<2>(src, srcStride, in + 3);
+
+            int16x4_t res2;
+            filter4_ss_s16x4<coeff4, shift>(in + 0, filter, c, res0);
+            filter4_ss_s16x4<coeff4, shift>(in + 1, filter, c, res1);
+
+            store_s16xnxm<n_store, 2>(res, dst, dstStride);
         }
     }
     else
     {
-        const int n_store = (width < 8) ? width : 8;
         for (int col = 0; col < width; col += 8)
         {
             const int16_t *s = src;
@@ -244,32 +263,17 @@
             load_s16x8xn<3>(s, srcStride, in);
             s += 3 * srcStride;
 
-            for (int row = 0; (row + 4) <= height; row += 4)
+            for (int row = 0; row + 4 <= height; row += 4)
             {
                 load_s16x8xn<4>(s, srcStride, in + 3);
 
-                int32x4_t sum_lo4;
-                int32x4_t sum_hi4;
-                filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0,
-                              sum_hi0);
-                filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1,
-                              sum_hi1);
-                filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2,
-                              sum_hi2);
-                filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3,
-                              sum_hi3);
-
-                int16x8_t sum4;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi0, IF_FILTER_PREC));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi1, IF_FILTER_PREC));
-                sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi2, IF_FILTER_PREC));
-                sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi3, IF_FILTER_PREC));
+                int16x8_t res4;
+                filter4_ss_s16x8<coeff4, shift>(in + 0, filter, c, res0);
+                filter4_ss_s16x8<coeff4, shift>(in + 1, filter, c, res1);
+                filter4_ss_s16x8<coeff4, shift>(in + 2, filter, c, res2);
+                filter4_ss_s16x8<coeff4, shift>(in + 3, filter, c, res3);
 
-                store_s16xnxm<n_store, 4>(sum, d, dstStride);
+                store_s16x8xn<4>(d, dstStride, res);
 
                 in0 = in4;
                 in1 = in5;
@@ -283,20 +287,11 @@
             {
                 load_s16x8xn<2>(s, srcStride, in + 3);
 
-                int32x4_t sum_lo2;
-                int32x4_t sum_hi2;
-                filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0,
-                              sum_hi0);
-                filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1,
-                              sum_hi1);
-
-                int16x8_t sum2;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi0, IF_FILTER_PREC));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, IF_FILTER_PREC),
-                                      vshrn_n_s32(sum_hi1, IF_FILTER_PREC));
+                int16x8_t res2;
+                filter4_ss_s16x8<coeff4, shift>(in + 0, filter, c, res0);
+                filter4_ss_s16x8<coeff4, shift>(in + 1, filter, c, res1);
 
-                store_s16xnxm<n_store, 2>(sum, d, dstStride);
+                store_s16x8xn<2>(d, dstStride, res);
             }
 
             src += 8;
@@ -310,6 +305,8 @@
                           intptr_t dstStride)
 {
     const int N_TAPS = 8;
+    const int16x8_t filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdx);
+
     src -= (N_TAPS / 2 - 1) * srcStride;
 
     // Zero constant in order to use filter helper functions (optimised away).
@@ -331,10 +328,10 @@
 
                 int32x4_t sum_lo4;
                 int32x4_t sum_hi4;
-                filter8_s16x8<coeffIdx>(in + 0, c, sum_lo0, sum_hi0);
-                filter8_s16x8<coeffIdx>(in + 1, c, sum_lo1, sum_hi1);
-                filter8_s16x8<coeffIdx>(in + 2, c, sum_lo2, sum_hi2);
-                filter8_s16x8<coeffIdx>(in + 3, c, sum_lo3, sum_hi3);
+                filter8_s16x8<coeffIdx>(in + 0, filter, c, sum_lo0, sum_hi0);
+                filter8_s16x8<coeffIdx>(in + 1, filter, c, sum_lo1, sum_hi1);
+                filter8_s16x8<coeffIdx>(in + 2, filter, c, sum_lo2, sum_hi2);
+                filter8_s16x8<coeffIdx>(in + 3, filter, c, sum_lo3, sum_hi3);
 
                 int16x8_t sum4;
                 sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
@@ -373,10 +370,10 @@
             load_s16x4xn<4>(s, srcStride, in + 7);
 
             int32x4_t sum4;
-            filter8_s16x4<coeffIdx>(in + 0, c, sum0);
-            filter8_s16x4<coeffIdx>(in + 1, c, sum1);
-            filter8_s16x4<coeffIdx>(in + 2, c, sum2);
-            filter8_s16x4<coeffIdx>(in + 3, c, sum3);
+            filter8_s16x4<coeffIdx>(in + 0, filter, c, sum0);
+            filter8_s16x4<coeffIdx>(in + 1, filter, c, sum1);
+            filter8_s16x4<coeffIdx>(in + 2, filter, c, sum2);
+            filter8_s16x4<coeffIdx>(in + 3, filter, c, sum3);
 
             int16x4_t sum_s164;
             sum_s160 = vshrn_n_s32(sum0, IF_FILTER_PREC);
@@ -415,10 +412,10 @@
 
                 int32x4_t sum_lo4;
                 int32x4_t sum_hi4;
-                filter8_s16x8<coeffIdx>(in + 0, c, sum_lo0, sum_hi0);
-                filter8_s16x8<coeffIdx>(in + 1, c, sum_lo1, sum_hi1);
-                filter8_s16x8<coeffIdx>(in + 2, c, sum_lo2, sum_hi2);
-                filter8_s16x8<coeffIdx>(in + 3, c, sum_lo3, sum_hi3);
+                filter8_s16x8<coeffIdx>(in + 0, filter, c, sum_lo0, sum_hi0);
+                filter8_s16x8<coeffIdx>(in + 1, filter, c, sum_lo1, sum_hi1);
+                filter8_s16x8<coeffIdx>(in + 2, filter, c, sum_lo2, sum_hi2);
+                filter8_s16x8<coeffIdx>(in + 3, filter, c, sum_lo3, sum_hi3);
 
                 int16x8_t sum4;
                 sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, IF_FILTER_PREC),
@@ -1690,116 +1687,119 @@
     }
 }
 
-template<int width, int height>
+template<bool coeff4, int width, int height>
 void interp4_vert_sp_neon(const int16_t *src, intptr_t srcStride, uint8_t *dst,
                           intptr_t dstStride, int coeffIdx)
 {
     assert(X265_DEPTH == 8);
+    const int N_TAPS = 4;
     const int headRoom = IF_INTERNAL_PREC - X265_DEPTH;
     const int shift = IF_FILTER_PREC + headRoom;
-    const int offset = (1 << (shift - 1)) + (IF_INTERNAL_OFFS <<
-        IF_FILTER_PREC);
-
-    const int N_TAPS = 4;
-    src -= (N_TAPS / 2 - 1) * srcStride;
+    // Subtract 8 from shift since we account for that in table lookups.
+    const int shift_offset = shift - 8;
 
     const int16x4_t filter = vld1_s16(X265_NS::g_chromaFiltercoeffIdx);
-    const int32x4_t c = vdupq_n_s32(offset);
+    const uint8x16_t shr_tbl = vld1q_u8(vert_shr_tbl);
+    int32x4_t offset;
 
-    if (width == 12)
+    if (coeff4)
     {
-        const int16_t *s = src;
-        uint8_t *d = dst;
+        // The right shift by 2 is needed because we will divide the filter values by 4.
+        offset = vdupq_n_s32(((1 << (shift - 1)) +
+                              (IF_INTERNAL_OFFS << IF_FILTER_PREC)) >> 2);
+    }
+    else
+    {
+        offset = vdupq_n_s32((1 << (shift - 1)) +
+                             (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+    }
 
-        int16x8_t in7;
-        load_s16x8xn<3>(s, srcStride, in);
-        s += 3 * srcStride;
+    src -= (N_TAPS / 2 - 1) * srcStride;
 
-        for (int row = 0; (row + 4) <= height; row += 4)
+    if (width % 8 != 0)
+    {
+        if (width == 12 || width == 6)
         {
-            load_s16x8xn<4>(s, srcStride, in + 3);
+            const int n_store = width == 12 ? 8 : 6;
+            const int16_t *s = src;
+            uint8_t *d = dst;
+
+            int16x8_t in7;
+            load_s16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
 
-            int32x4_t sum_lo4;
-            int32x4_t sum_hi4;
-            filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0, sum_hi0);
-            filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1, sum_hi1);
-            filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2, sum_hi2);
-            filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3, sum_hi3);
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 3);
 
-            int16x8_t sum4;
-            sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                  vshrn_n_s32(sum_hi0, shift));
-            sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                  vshrn_n_s32(sum_hi1, shift));
-            sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, shift),
-                                  vshrn_n_s32(sum_hi2, shift));
-            sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, shift),
-                                  vshrn_n_s32(sum_hi3, shift));
+                uint8x8_t sum4;
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 0, filter, offset, shr_tbl,
+                                                       sum0);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 1, filter, offset, shr_tbl,
+                                                       sum1);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 2, filter, offset, shr_tbl,
+                                                       sum2);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 3, filter, offset, shr_tbl,
+                                                       sum3);
 
-            uint8x8_t sum_u84;
-            sum_u80 = vqmovun_s16(sum0);
-            sum_u81 = vqmovun_s16(sum1);
-            sum_u82 = vqmovun_s16(sum2);
-            sum_u83 = vqmovun_s16(sum3);
+                store_u8xnxm<n_store, 4>(d, dstStride, sum);
 
-            store_u8x8xn<4>(d, dstStride, sum_u8);
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
 
-            in0 = in4;
-            in1 = in5;
-            in2 = in6;
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
 
-            s += 4 * srcStride;
-            d += 4 * dstStride;
+            if (width == 6)
+            {
+                return;
+            }
+
+            src += 8;
+            dst += 8;
         }
 
-        src += 8;
-        dst += 8;
-        s = src;
-        d = dst;
+        const int n_store = width > 4 ? 4 : width;
 
-        load_s16x8xn<3>(s, srcStride, in);
-        s += 3 * srcStride;
+        int16x4_t in7;
+        load_s16x4xn<3>(src, srcStride, in);
+        src += 3 * srcStride;
 
-        for (int row = 0; (row + 4) <= height; row += 4)
+        for (int row = 0; row + 4 <= height; row += 4)
         {
-            load_s16x8xn<4>(s, srcStride, in + 3);
-
-            int32x4_t sum_lo4;
-            int32x4_t sum_hi4;
-            filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0, sum_hi0);
-            filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1, sum_hi1);
-            filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2, sum_hi2);
-            filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3, sum_hi3);
-
-            int16x8_t sum4;
-            sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                  vshrn_n_s32(sum_hi0, shift));
-            sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                  vshrn_n_s32(sum_hi1, shift));
-            sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, shift),
-                                  vshrn_n_s32(sum_hi2, shift));
-            sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, shift),
-                                  vshrn_n_s32(sum_hi3, shift));
+            load_s16x4xn<4>(src, srcStride, in + 3);
 
-            uint8x8_t sum_u84;
-            sum_u80 = vqmovun_s16(sum0);
-            sum_u81 = vqmovun_s16(sum1);
-            sum_u82 = vqmovun_s16(sum2);
-            sum_u83 = vqmovun_s16(sum3);
+            uint8x8_t res2;
+            filter4x2_sp_s16x4<coeff4, shift_offset>(in + 0, in + 1, filter, offset,
+                                                     shr_tbl, res0);
+            filter4x2_sp_s16x4<coeff4, shift_offset>(in + 2, in + 3, filter, offset,
+                                                     shr_tbl, res1);
 
-            store_u8x4xn<4>(d, dstStride, sum_u8);
+            store_u8xnxm_strided<n_store, 4>(dst, dstStride, res);
 
             in0 = in4;
             in1 = in5;
             in2 = in6;
 
-            s += 4 * srcStride;
-            d += 4 * dstStride;
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_s16x4xn<2>(src, srcStride, in + 3);
+
+            uint8x8_t res;
+            filter4x2_sp_s16x4<coeff4, shift_offset>(in + 0, in + 1, filter, offset,
+                                                     shr_tbl, res);
+
+            store_u8xnxm_strided<n_store, 2>(dst, dstStride, &res);
         }
     }
     else
     {
-        const int n_store = (width < 8) ? width : 8;
         for (int col = 0; col < width; col += 8)
         {
             const int16_t *s = src;
@@ -1809,38 +1809,21 @@
             load_s16x8xn<3>(s, srcStride, in);
             s += 3 * srcStride;
 
-            for (int row = 0; (row + 4) <= height; row += 4)
+            for (int row = 0; row + 4 <= height; row += 4)
             {
                 load_s16x8xn<4>(s, srcStride, in + 3);
 
-                int32x4_t sum_lo4;
-                int32x4_t sum_hi4;
-                filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0,
-                              sum_hi0);
-                filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1,
-                              sum_hi1);
-                filter4_s16x8(coeffIdx, in + 2, filter, c, sum_lo2,
-                              sum_hi2);
-                filter4_s16x8(coeffIdx, in + 3, filter, c, sum_lo3,
-                              sum_hi3);
-
-                int16x8_t sum4;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                      vshrn_n_s32(sum_hi0, shift));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                      vshrn_n_s32(sum_hi1, shift));
-                sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, shift),
-                                      vshrn_n_s32(sum_hi2, shift));
-                sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, shift),
-                                      vshrn_n_s32(sum_hi3, shift));
+                uint8x8_t sum4;
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 0, filter, offset, shr_tbl,
+                                                       sum0);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 1, filter, offset, shr_tbl,
+                                                       sum1);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 2, filter, offset, shr_tbl,
+                                                       sum2);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 3, filter, offset, shr_tbl,
+                                                       sum3);
 
-                uint8x8_t sum_u84;
-                sum_u80 = vqmovun_s16(sum0);
-                sum_u81 = vqmovun_s16(sum1);
-                sum_u82 = vqmovun_s16(sum2);
-                sum_u83 = vqmovun_s16(sum3);
-
-                store_u8xnxm<n_store, 4>(d, dstStride, sum_u8);
+                store_u8x8xn<4>(d, dstStride, sum);
 
                 in0 = in4;
                 in1 = in5;
@@ -1854,24 +1837,13 @@
             {
                 load_s16x8xn<2>(s, srcStride, in + 3);
 
-                int32x4_t sum_lo2;
-                int32x4_t sum_hi2;
-                filter4_s16x8(coeffIdx, in + 0, filter, c, sum_lo0,
-                              sum_hi0);
-                filter4_s16x8(coeffIdx, in + 1, filter, c, sum_lo1,
-                              sum_hi1);
-
-                int16x8_t sum2;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                      vshrn_n_s32(sum_hi0, shift));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                      vshrn_n_s32(sum_hi1, shift));
+                uint8x8_t sum2;
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 0, filter, offset, shr_tbl,
+                                                       sum0);
+                filter4_sp_s16x8<coeff4, shift_offset>(in + 1, filter, offset, shr_tbl,
+                                                       sum1);
 
-                uint8x8_t sum_u82;
-                sum_u80 = vqmovun_s16(sum0);
-                sum_u81 = vqmovun_s16(sum1);
-
-                store_u8xnxm<n_store, 2>(d, dstStride, sum_u8);
+                store_u8x8xn<2>(d, dstStride, sum);
             }
 
             src += 8;
@@ -1887,13 +1859,16 @@
     assert(X265_DEPTH == 8);
     const int headRoom = IF_INTERNAL_PREC - X265_DEPTH;
     const int shift = IF_FILTER_PREC + headRoom;
-    const int offset = (1 << (shift - 1)) + (IF_INTERNAL_OFFS <<
-        IF_FILTER_PREC);
+    // Subtract 8 from shift since we account for that in table lookups.
+    const int shift_offset = shift - 8;
+    const int offset = (1 << (shift - 1)) + (IF_INTERNAL_OFFS << IF_FILTER_PREC);
 
     const int N_TAPS = 8;
     src -= (N_TAPS / 2 - 1) * srcStride;
 
+    const int16x8_t filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdx);
     const int32x4_t c = vdupq_n_s32(offset);
+    const uint8x16_t shr_tbl = vld1q_u8(vert_shr_tbl);
 
     if (width % 8 != 0)
     {
@@ -1910,28 +1885,23 @@
             {
                 load_s16x8xn<4>(s, srcStride, in + 7);
 
-                int32x4_t sum_lo4;
-                int32x4_t sum_hi4;
-                filter8_s16x8<coeffIdx>(in + 0, c, sum_lo0, sum_hi0);
-                filter8_s16x8<coeffIdx>(in + 1, c, sum_lo1, sum_hi1);
-                filter8_s16x8<coeffIdx>(in + 2, c, sum_lo2, sum_hi2);
-                filter8_s16x8<coeffIdx>(in + 3, c, sum_lo3, sum_hi3);
+                int32x4_t sum_lo4, sum_hi4;
+                filter8_s16x8<coeffIdx>(in + 0, filter, c, sum_lo0, sum_hi0);
+                filter8_s16x8<coeffIdx>(in + 1, filter, c, sum_lo1, sum_hi1);
+                filter8_s16x8<coeffIdx>(in + 2, filter, c, sum_lo2, sum_hi2);
+                filter8_s16x8<coeffIdx>(in + 3, filter, c, sum_lo3, sum_hi3);
 
                 int16x8_t sum4;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                      vshrn_n_s32(sum_hi0, shift));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                      vshrn_n_s32(sum_hi1, shift));
-                sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, shift),
-                                      vshrn_n_s32(sum_hi2, shift));
-                sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, shift),
-                                      vshrn_n_s32(sum_hi3, shift));
+                sum0 = vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl);
+                sum1 = vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl);
+                sum2 = vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl);
+                sum3 = vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl);
 
                 uint8x8_t sum_u84;
-                sum_u80 = vqmovun_s16(sum0);
-                sum_u81 = vqmovun_s16(sum1);
-                sum_u82 = vqmovun_s16(sum2);
-                sum_u83 = vqmovun_s16(sum3);
+                sum_u80 = vqshrun_n_s16(sum0, shift_offset);
+                sum_u81 = vqshrun_n_s16(sum1, shift_offset);
+                sum_u82 = vqshrun_n_s16(sum2, shift_offset);
+                sum_u83 = vqshrun_n_s16(sum3, shift_offset);
 
                 store_u8x8xn<4>(d, dstStride, sum_u8);
 
@@ -1960,24 +1930,20 @@
             load_s16x4xn<4>(s, srcStride, in + 7);
 
             int32x4_t sum4;
-            filter8_s16x4<coeffIdx>(in + 0, c, sum0);
-            filter8_s16x4<coeffIdx>(in + 1, c, sum1);
-            filter8_s16x4<coeffIdx>(in + 2, c, sum2);
-            filter8_s16x4<coeffIdx>(in + 3, c, sum3);
-
-            int16x4_t sum_s164;
-            sum_s160 = vshrn_n_s32(sum0, shift);
-            sum_s161 = vshrn_n_s32(sum1, shift);
-            sum_s162 = vshrn_n_s32(sum2, shift);
-            sum_s163 = vshrn_n_s32(sum3, shift);
+            filter8_s16x4<coeffIdx>(in + 0, filter, c, sum0);
+            filter8_s16x4<coeffIdx>(in + 1, filter, c, sum1);
+            filter8_s16x4<coeffIdx>(in + 2, filter, c, sum2);
+            filter8_s16x4<coeffIdx>(in + 3, filter, c, sum3);
+
+            int16x8_t sum_s162;
+            sum_s160 = vtbl2q_s32_s16(sum0, sum1, shr_tbl);
+            sum_s161 = vtbl2q_s32_s16(sum2, sum3, shr_tbl);
+
+            uint8x8_t sum_u82;
+            sum_u80 = vqshrun_n_s16(sum_s160, shift_offset);
+            sum_u81 = vqshrun_n_s16(sum_s161, shift_offset);
 
-            uint8x8_t sum_u84;
-            sum_u80 = vqmovun_s16(vcombine_s16(sum_s160, vdup_n_s16(0)));
-            sum_u81 = vqmovun_s16(vcombine_s16(sum_s161, vdup_n_s16(0)));
-            sum_u82 = vqmovun_s16(vcombine_s16(sum_s162, vdup_n_s16(0)));
-            sum_u83 = vqmovun_s16(vcombine_s16(sum_s163, vdup_n_s16(0)));
-
-            store_u8x4xn<4>(d, dstStride, sum_u8);
+            store_u8x4_strided_xN<4>(d, dstStride, sum_u8);
 
             in0 = in4;
             in1 = in5;
@@ -2006,28 +1972,23 @@
             {
                 load_s16x8xn<4>(s, srcStride, in + 7);
 
-                int32x4_t sum_lo4;
-                int32x4_t sum_hi4;
-                filter8_s16x8<coeffIdx>(in + 0, c, sum_lo0, sum_hi0);
-                filter8_s16x8<coeffIdx>(in + 1, c, sum_lo1, sum_hi1);
-                filter8_s16x8<coeffIdx>(in + 2, c, sum_lo2, sum_hi2);
-                filter8_s16x8<coeffIdx>(in + 3, c, sum_lo3, sum_hi3);
+                int32x4_t sum_lo4, sum_hi4;
+                filter8_s16x8<coeffIdx>(in + 0, filter, c, sum_lo0, sum_hi0);
+                filter8_s16x8<coeffIdx>(in + 1, filter, c, sum_lo1, sum_hi1);
+                filter8_s16x8<coeffIdx>(in + 2, filter, c, sum_lo2, sum_hi2);
+                filter8_s16x8<coeffIdx>(in + 3, filter, c, sum_lo3, sum_hi3);
 
                 int16x8_t sum4;
-                sum0 = vcombine_s16(vshrn_n_s32(sum_lo0, shift),
-                                      vshrn_n_s32(sum_hi0, shift));
-                sum1 = vcombine_s16(vshrn_n_s32(sum_lo1, shift),
-                                      vshrn_n_s32(sum_hi1, shift));
-                sum2 = vcombine_s16(vshrn_n_s32(sum_lo2, shift),
-                                      vshrn_n_s32(sum_hi2, shift));
-                sum3 = vcombine_s16(vshrn_n_s32(sum_lo3, shift),
-                                      vshrn_n_s32(sum_hi3, shift));
+                sum0 = vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl);
+                sum1 = vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl);
+                sum2 = vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl);
+                sum3 = vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl);
 
                 uint8x8_t sum_u84;
-                sum_u80 = vqmovun_s16(sum0);
-                sum_u81 = vqmovun_s16(sum1);
-                sum_u82 = vqmovun_s16(sum2);
-                sum_u83 = vqmovun_s16(sum3);
+                sum_u80 = vqshrun_n_s16(sum0, shift_offset);
+                sum_u81 = vqshrun_n_s16(sum1, shift_offset);
+                sum_u82 = vqshrun_n_s16(sum2, shift_offset);
+                sum_u83 = vqshrun_n_s16(sum3, shift_offset);
 
                 store_u8x8xn<4>(d, dstStride, sum_u8);
 
@@ -2049,66 +2010,911 @@
     }
 }
 
-#endif // !HIGH_BIT_DEPTH
+template<int coeffIdx, int coeffIdy, int width, int height>
+void interp8_hv_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                        intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    const int v_shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    // Subtract 8 from shift since we account for that in table lookups.
+    const int v_shift_offset = v_shift - 8;
+    const int16x8_t v_filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdy);
+    const uint16x8_t h_offset = vdupq_n_u16((uint16_t)-IF_INTERNAL_OFFS);
+    const int32x4_t v_offset = vdupq_n_s32((1 << (v_shift - 1)) +
+                                           (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+    const uint8x16_t shr_tbl = vld1q_u8(vert_shr_tbl);
+
+    src -= (N_TAPS / 2 - 1) * srcStride + (N_TAPS / 2 - 1);
+
+    int col = 0;
+    for (; col + 16 <= width; col += 16)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        uint8x16_t h_sN_TAPS;
+        int16x8_t v_s011, v_s111;
+
+        load_u8x16xn<8>(s + 0 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s00, v_s10);
+
+        load_u8x16xn<8>(s + 1 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s01, v_s11);
+
+        load_u8x16xn<8>(s + 2 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s02, v_s12);
+
+        load_u8x16xn<8>(s + 3 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s03, v_s13);
+
+        load_u8x16xn<8>(s + 4 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s04, v_s14);
+
+        load_u8x16xn<8>(s + 5 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s05, v_s15);
+
+        load_u8x16xn<8>(s + 6 * srcStride, 1, h_s);
+        filter8_u8x16<coeffIdx>(h_s, h_offset, v_s06, v_s16);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            uint8x8_t res_lo4, res_hi4;
+            int32x4_t sum_lo8, sum_hi8;
+
+            load_u8x16xn<8>(s + 0 * srcStride, 1, h_s);
+            filter8_u8x16<coeffIdx>(h_s, h_offset, v_s07, v_s17);
+            filter8_s16x8<coeffIdy>(v_s0 + 0, v_filter, v_offset, sum_lo0, sum_hi0);
+            filter8_s16x8<coeffIdy>(v_s1 + 0, v_filter, v_offset, sum_lo1, sum_hi1);
+            v_s00 = v_s04;
+            v_s10 = v_s14;
+            res_lo0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl),
+                                      v_shift_offset);
+            res_hi0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl),
+                                      v_shift_offset);
+
+            load_u8x16xn<8>(s + 1 * srcStride, 1, h_s);
+            filter8_u8x16<coeffIdx>(h_s, h_offset, v_s08, v_s18);
+            filter8_s16x8<coeffIdy>(v_s0 + 1, v_filter, v_offset, sum_lo2, sum_hi2);
+            filter8_s16x8<coeffIdy>(v_s1 + 1, v_filter, v_offset, sum_lo3, sum_hi3);
+            v_s01 = v_s05;
+            v_s11 = v_s15;
+            res_lo1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl),
+                                      v_shift_offset);
+            res_hi1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl),
+                                      v_shift_offset);
+
+            load_u8x16xn<8>(s + 2 * srcStride, 1, h_s);
+            filter8_u8x16<coeffIdx>(h_s, h_offset, v_s09, v_s19);
+            filter8_s16x8<coeffIdy>(v_s0 + 2, v_filter, v_offset, sum_lo4, sum_hi4);
+            filter8_s16x8<coeffIdy>(v_s1 + 2, v_filter, v_offset, sum_lo5, sum_hi5);
+            v_s02 = v_s06;
+            v_s12 = v_s16;
+            res_lo2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo4, sum_hi4, shr_tbl),
+                                      v_shift_offset);
+            res_hi2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo5, sum_hi5, shr_tbl),
+                                      v_shift_offset);
+
+            load_u8x16xn<8>(s + 3 * srcStride, 1, h_s);
+            filter8_u8x16<coeffIdx>(h_s, h_offset, v_s010, v_s110);
+            filter8_s16x8<coeffIdy>(v_s0 + 3, v_filter, v_offset, sum_lo6, sum_hi6);
+            filter8_s16x8<coeffIdy>(v_s1 + 3, v_filter, v_offset, sum_lo7, sum_hi7);
+            v_s03 = v_s07;
+            v_s13 = v_s17;
+            res_lo3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo6, sum_hi6, shr_tbl),
+                                      v_shift_offset);
+            res_hi3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo7, sum_hi7, shr_tbl),
+                                      v_shift_offset);
+
+            vst1q_u8(d + 0 * dstStride, vcombine_u8(res_lo0, res_hi0));
+            vst1q_u8(d + 1 * dstStride, vcombine_u8(res_lo1, res_hi1));
+            vst1q_u8(d + 2 * dstStride, vcombine_u8(res_lo2, res_hi2));
+            vst1q_u8(d + 3 * dstStride, vcombine_u8(res_lo3, res_hi3));
+
+            v_s04 = v_s08;
+            v_s14 = v_s18;
+            v_s05 = v_s09;
+            v_s15 = v_s19;
+            v_s06 = v_s010;
+            v_s16 = v_s110;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += 16;
+        dst += 16;
+    }
+
+    for (; col + 8 <= width; col += 8)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        uint8x8_t h_sN_TAPS;
+        int16x8_t v_s11;
+
+        load_u8x8xn<8>(s + 0 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s0);
+
+        load_u8x8xn<8>(s + 1 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s1);
+
+        load_u8x8xn<8>(s + 2 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s2);
+
+        load_u8x8xn<8>(s + 3 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s3);
+
+        load_u8x8xn<8>(s + 4 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s4);
+
+        load_u8x8xn<8>(s + 5 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s5);
+
+        load_u8x8xn<8>(s + 6 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, v_s6);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            uint8x8_t res4;
+            int32x4_t sum_lo4, sum_hi4;
+
+            load_u8x8xn<8>(s + 0 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, v_s7);
+            filter8_s16x8<coeffIdy>(v_s + 0, v_filter, v_offset, sum_lo0, sum_hi0);
+
+            load_u8x8xn<8>(s + 1 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, v_s8);
+            filter8_s16x8<coeffIdy>(v_s + 1, v_filter, v_offset, sum_lo1, sum_hi1);
+            v_s0 = v_s4;
+            v_s1 = v_s5;
+            res0 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo0, sum_hi0, shr_tbl),
+                                   v_shift_offset);
+            res1 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo1, sum_hi1, shr_tbl),
+                                   v_shift_offset);
+
+            load_u8x8xn<8>(s + 2 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, v_s9);
+            filter8_s16x8<coeffIdy>(v_s + 2, v_filter, v_offset, sum_lo2, sum_hi2);
+
+            load_u8x8xn<8>(s + 3 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, v_s10);
+            filter8_s16x8<coeffIdy>(v_s + 3, v_filter, v_offset, sum_lo3, sum_hi3);
+            v_s2 = v_s6;
+            v_s3 = v_s7;
+            res2 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo2, sum_hi2, shr_tbl),
+                                   v_shift_offset);
+            res3 = vqshrun_n_s16(vtbl2q_s32_s16(sum_lo3, sum_hi3, shr_tbl),
+                                   v_shift_offset);
+
+            store_u8xnxm<8, 4>(d + 0, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += 8;
+        dst += 8;
+    }
+
+    if (width % 8 != 0)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        uint8x8_t h_sN_TAPS;
+        int16x8_t t_v_s11;
+        int16x4_t v_s11;
+
+        load_u8x8xn<8>(s + 0 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s0);
+        v_s0 = vget_low_s16(t_v_s0);
+
+        load_u8x8xn<8>(s + 1 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s1);
+        v_s1 = vget_low_s16(t_v_s1);
+
+        load_u8x8xn<8>(s + 2 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s2);
+        v_s2 = vget_low_s16(t_v_s2);
+
+        load_u8x8xn<8>(s + 3 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s3);
+        v_s3 = vget_low_s16(t_v_s3);
+
+        load_u8x8xn<8>(s + 4 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s4);
+        v_s4 = vget_low_s16(t_v_s4);
+
+        load_u8x8xn<8>(s + 5 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s5);
+        v_s5 = vget_low_s16(t_v_s5);
+
+        load_u8x8xn<8>(s + 6 * srcStride, 1, h_s);
+        filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s6);
+        v_s6 = vget_low_s16(t_v_s6);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            int32x4_t sum4;
+            uint8x8_t res2;
+
+            load_u8x8xn<8>(s + 0 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s7);
+            v_s7 = vget_low_s16(t_v_s7);
+            filter8_s16x4<coeffIdy>(v_s + 0, v_filter, v_offset, sum0);
+            v_s0 = v_s4;
+
+            load_u8x8xn<8>(s + 1 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s8);
+            v_s8 = vget_low_s16(t_v_s8);
+            filter8_s16x4<coeffIdy>(v_s + 1, v_filter, v_offset, sum1);
+            v_s1 = v_s5;
+
+            load_u8x8xn<8>(s + 2 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s9);
+            v_s9 = vget_low_s16(t_v_s9);
+            filter8_s16x4<coeffIdy>(v_s + 2, v_filter, v_offset, sum2);
+            v_s2 = v_s6;
+
+            load_u8x8xn<8>(s + 3 * srcStride, 1, h_s);
+            filter8_u8x8<coeffIdx>(h_s, h_offset, t_v_s10);
+            v_s10 = vget_low_s16(t_v_s10);
+            filter8_s16x4<coeffIdy>(v_s + 3, v_filter, v_offset, sum3);
+            v_s3 = v_s7;
+
+            res0 = vqshrun_n_s16(vtbl2q_s32_s16(sum0, sum1, shr_tbl),
+                                   v_shift_offset);
+            res1 = vqshrun_n_s16(vtbl2q_s32_s16(sum2, sum3, shr_tbl),
+                                   v_shift_offset);
+
+            store_u8x4_strided_xN<4>(d + 0 * dstStride, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+    }
 }
 
-namespace X265_NS
+#else // !HIGH_BIT_DEPTH
+
+#if X265_DEPTH == 10
+template<bool coeff4>
+void inline filter4_u16x4(const uint16x4_t *s, uint16x4_t f,
+                          const uint16x8_t offset, const uint16x4_t maxVal,
+                          uint16x4_t &d)
 {
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x4_t sum03 = vadd_u16(s0, s3);
+        uint16x4_t sum12 = vadd_u16(s1, s2);
+
+        int16x4_t sum =
+            vreinterpret_s16_u16(vmla_n_u16(vget_low_u16(offset), sum12, 9));
+        sum = vsub_s16(sum, vreinterpret_s16_u16(sum03));
 
-#if HIGH_BIT_DEPTH
-#define SHIFT_INTERP_PS (IF_FILTER_PREC - (IF_INTERNAL_PREC - X265_DEPTH))
-#endif
+        // We divided filter values by 4 so -2 from right shift.
+        sum = vshr_n_s16(sum, IF_FILTER_PREC - 2);
 
-template<int width, int height>
-void filterPixelToShort_neon(const pixel *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride)
+        d = vreinterpret_u16_s16(vmax_s16(sum, vdup_n_s16(0)));
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        // All chroma filter taps have signs {-, +, +, -}, so we can use a
+        // sequence of MLA/MLS with absolute filter values to avoid needing to
+        // widen the input.
+
+        uint16x4_t sum01 = vmul_lane_u16(s1, f, 1);
+        sum01 = vmls_lane_u16(sum01, s0, f, 0);
+
+        uint16x4_t sum23 = vmla_lane_u16(vget_low_u16(offset), s2, f, 2);
+        sum23 = vmls_lane_u16(sum23, s3, f, 3);
+
+        int32x4_t sum = vaddl_s16(vreinterpret_s16_u16(sum01),
+                                  vreinterpret_s16_u16(sum23));
+
+        // We halved filter values so -1 from right shift.
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC - 1);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<bool coeff4>
+void inline filter4_u16x8(const uint16x8_t *s, uint16x4_t f,
+                          const uint16x8_t offset, const uint16x8_t maxVal,
+                          uint16x8_t &d)
 {
-    const int shift = IF_INTERNAL_PREC - X265_DEPTH;
-    const int16x8_t off = vdupq_n_s16(IF_INTERNAL_OFFS);
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x8_t sum03 = vaddq_u16(s0, s3);
+        uint16x8_t sum12 = vaddq_u16(s1, s2);
+
+        int16x8_t sum = vreinterpretq_s16_u16(vmlaq_n_u16(offset, sum12, 9));
+        sum = vsubq_s16(sum, vreinterpretq_s16_u16(sum03));
+
+        // We divided filter values by 4 so -2 from right shift.
+        sum = vshrq_n_s16(sum, IF_FILTER_PREC - 2);
+
+        d = vreinterpretq_u16_s16(vmaxq_s16(sum, vdupq_n_s16(0)));
+        d = vminq_u16(d, maxVal);
+    }
+    else
+    {
+        // All chroma filter taps have signs {-, +, +, -}, so we can use a
+        // sequence of MLA/MLS with absolute filter values to avoid needing to
+        // widen the input.
+        uint16x8_t sum01 = vmulq_lane_u16(s1, f, 1);
+        sum01 = vmlsq_lane_u16(sum01, s0, f, 0);
+
+        uint16x8_t sum23 = vmlaq_lane_u16(offset, s2, f, 2);
+        sum23 = vmlsq_lane_u16(sum23, s3, f, 3);
+
+        int32x4_t sum_lo = vaddl_s16(
+            vreinterpret_s16_u16(vget_low_u16(sum01)),
+            vreinterpret_s16_u16(vget_low_u16(sum23)));
+        int32x4_t sum_hi = vaddl_s16(
+            vreinterpret_s16_u16(vget_high_u16(sum01)),
+            vreinterpret_s16_u16(vget_high_u16(sum23)));
+
+        // We halved filter values so -1 from right shift.
+        uint16x4_t d0 = vqshrun_n_s32(sum_lo, IF_FILTER_PREC - 1);
+        uint16x4_t d1 = vqshrun_n_s32(sum_hi, IF_FILTER_PREC - 1);
+
+        d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+    }
+}
+
+#else // X265_DEPTH == 12
+template<bool coeff4>
+void inline filter4_u16x4(const uint16x4_t *s, const uint16x4_t f,
+                          const uint32x4_t offset, const uint16x4_t maxVal,
+                          uint16x4_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x4_t sum03 = vadd_u16(s0, s3);
+        uint16x4_t sum12 = vadd_u16(s1, s2);
+
+        int32x4_t sum = vreinterpretq_s32_u32(vmlal_n_u16(offset, sum12, 9));
+        sum = vsubw_s16(sum, vreinterpret_s16_u16(sum03));
+
+        // We divided filter values by 4 so -2 from right shift.
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC - 2);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        uint32x4_t sum = vmlsl_lane_u16(offset, s0, f, 0);
+        sum = vmlal_lane_u16(sum, s1, f, 1);
+        sum = vmlal_lane_u16(sum, s2, f, 2);
+        sum = vmlsl_lane_u16(sum, s3, f, 3);
+
+        d = vqshrun_n_s32(vreinterpretq_s32_u32(sum), IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<bool coeff4>
+void inline filter4_u16x8(const uint16x8_t *s, const uint16x4_t f,
+                          const uint32x4_t offset, const uint16x8_t maxVal,
+                          uint16x8_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x8_t sum03 = vaddq_u16(s0, s3);
+        uint16x8_t sum12 = vaddq_u16(s1, s2);
+
+        int32x4_t sum_lo = vreinterpretq_s32_u32(
+            vmlal_n_u16(offset, vget_low_u16(sum12), 9));
+        int32x4_t sum_hi = vreinterpretq_s32_u32(
+            vmlal_n_u16(offset, vget_high_u16(sum12), 9));
+        sum_lo = vsubw_s16(sum_lo, vreinterpret_s16_u16(vget_low_u16(sum03)));
+        sum_hi = vsubw_s16(sum_hi, vreinterpret_s16_u16(vget_high_u16(sum03)));
+
+        // We divided filter values by 4 so -2 from right shift.
+        uint16x4_t d0 = vqshrun_n_s32(sum_lo, IF_FILTER_PREC - 2);
+        uint16x4_t d1 = vqshrun_n_s32(sum_hi, IF_FILTER_PREC - 2);
+        d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+    }
+    else
+    {
+        uint32x4_t sum_lo = vmlsl_lane_u16(offset, vget_low_u16(s0), f, 0);
+        sum_lo = vmlal_lane_u16(sum_lo, vget_low_u16(s1), f, 1);
+        sum_lo = vmlal_lane_u16(sum_lo, vget_low_u16(s2), f, 2);
+        sum_lo = vmlsl_lane_u16(sum_lo, vget_low_u16(s3), f, 3);
+
+        uint32x4_t sum_hi = vmlsl_lane_u16(offset, vget_high_u16(s0), f, 0);
+        sum_hi = vmlal_lane_u16(sum_hi, vget_high_u16(s1), f, 1);
+        sum_hi = vmlal_lane_u16(sum_hi, vget_high_u16(s2), f, 2);
+        sum_hi = vmlsl_lane_u16(sum_hi, vget_high_u16(s3), f, 3);
+
+        uint16x4_t d0 = vqshrun_n_s32(vreinterpretq_s32_u32(sum_lo),
+                                      IF_FILTER_PREC);
+        uint16x4_t d1 = vqshrun_n_s32(vreinterpretq_s32_u32(sum_hi),
+                                      IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+    }
+}
+#endif // X265_DEPTH == 10
+
+template<bool coeff4, int width, int height>
+void inline interp4_horiz_pp_neon(const pixel *src, intptr_t srcStride,
+                                  pixel *dst, intptr_t dstStride,
+                                  const int16_t coeffIdx)
+{
+    const int N_TAPS = 4;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    uint16x4_t filter = vreinterpret_u16_s16(
+        vabs_s16(vld1_s16(X265_NS::g_chromaFiltercoeffIdx)));
+
+    uint16_t offset_u16;
+    // A shim of 1 << (IF_FILTER_PREC - 1) enables us to use non-rounding
+    // shifts - which are generally faster than rounding shifts on modern CPUs.
+    if (coeff4)
+    {
+        // The outermost -2 is needed because we will divide the filter values by 4.
+        offset_u16 = 1 << (IF_FILTER_PREC - 1 - 2);
+    }
+    else
+    {
+        offset_u16 = 1 << (IF_FILTER_PREC - 1);
+    }
+
+#if X265_DEPTH == 10
+    if (!coeff4)
+    {
+        // All filter values are even, halve them to avoid needing to widen to
+        // 32-bit elements in filter kernels.
+        filter = vshr_n_u16(filter, 1);
+        offset_u16 >>= 1;
+    }
+
+    const uint16x8_t offset = vdupq_n_u16(offset_u16);
+#else
+    const uint32x4_t offset = vdupq_n_u32(offset_u16);
+#endif // X265_DEPTH == 10
+
+    src -= N_TAPS / 2 - 1;
+
     for (int row = 0; row < height; row++)
     {
+        if (width % 16 == 0)
+        {
+            for (int col = 0; col < width; col += 16)
+            {
+                uint16x8_t s0N_TAPS, s1N_TAPS;
+                load_u16x8xn<4>(src + col + 0, 1, s0);
+                load_u16x8xn<4>(src + col + 8, 1, s1);
 
-        int col = 0;
-        for (; col + 8 <= width; col += 8)
+                uint16x8_t d0, d1;
+                filter4_u16x8<coeff4>(s0, filter, offset, maxVal, d0);
+                filter4_u16x8<coeff4>(s1, filter, offset, maxVal, d1);
+
+                vst1q_u16(dst + col + 0, d0);
+                vst1q_u16(dst + col + 8, d1);
+            }
+        }
+        else
         {
-            uint16x8_t in;
+            int col = 0;
+            for (; col + 8 <= width; col += 8)
+            {
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<4>(src + col, 1, s0);
 
-#if HIGH_BIT_DEPTH
-            in = vld1q_u16(src + col);
-#else
-            in = vmovl_u8(vld1_u8(src + col));
-#endif
+                uint16x8_t d0;
+                filter4_u16x8<coeff4>(s0, filter, offset, maxVal, d0);
 
-            int16x8_t tmp = vreinterpretq_s16_u16(vshlq_n_u16(in, shift));
-            tmp = vsubq_s16(tmp, off);
-            vst1q_s16(dst + col, tmp);
+                vst1q_u16(dst + col, d0);
+            }
+
+            if (width == 6)
+            {
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<4>(src, 1, s0);
+
+                uint16x8_t d0;
+                filter4_u16x8<coeff4>(s0, filter, offset, maxVal, d0);
+
+                store_u16x6xn<1>(dst, dstStride, &d0);
+            }
+            else if (width % 8 != 0)
+            {
+                uint16x4_t s0N_TAPS;
+                load_u16x4xn<4>(src + col, 1, s0);
+
+                uint16x4_t d0;
+                filter4_u16x4<coeff4>(s0, filter, offset,
+                                      vget_low_u16(maxVal), d0);
+
+                if (width == 2)
+                {
+                    store_u16x2xn<1>(dst + col, dstStride, &d0);
+                }
+                else
+                {
+                    vst1_u16(dst + col, d0);
+                }
+            }
         }
-        for (; col + 4 <= width; col += 4)
+
+        src += srcStride;
+        dst += dstStride;
+    }
+}
+
+#if X265_DEPTH == 10
+template<int coeffIdx>
+void inline filter8_u16x4(const uint16x4_t *s, uint16x4_t &d, uint16x8_t filter,
+                          uint16x4_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x4_t sum012456 = vsub_u16(s6, s0);
+        sum012456 = vmla_laneq_u16(sum012456, s1, filter, 1);
+        sum012456 = vmls_laneq_u16(sum012456, s2, filter, 2);
+        sum012456 = vmla_laneq_u16(sum012456, s4, filter, 4);
+        sum012456 = vmls_laneq_u16(sum012456, s5, filter, 5);
+
+        uint32x4_t sum3 = vmull_laneq_u16(s3, filter, 3);
+
+        int32x4_t d0 = vaddw_s16(vreinterpretq_s32_u32(sum3),
+                                 vreinterpret_s16_u16(sum012456));
+
+        d = vqrshrun_n_s32(d0, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x4_t sum07 = vadd_u16(s0, s7);
+        uint16x4_t sum16 = vadd_u16(s1, s6);
+        uint16x4_t sum25 = vadd_u16(s2, s5);
+        uint16x4_t sum34 = vadd_u16(s3, s4);
+
+        uint16x4_t sum0167 = vshl_n_u16(sum16, 2);
+        sum0167 = vsub_u16(sum0167, sum07);
+
+        uint32x4_t sum2345 = vmull_laneq_u16(sum34, filter, 3);
+        sum2345 = vmlsl_laneq_u16(sum2345, sum25, filter, 2);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum2345),
+                                  vreinterpret_s16_u16(sum0167));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x4_t sum123567 = vsub_u16(s1, s7);
+        sum123567 = vmls_laneq_u16(sum123567, s2, filter, 2);
+        sum123567 = vmla_laneq_u16(sum123567, s3, filter, 3);
+        sum123567 = vmla_laneq_u16(sum123567, s6, filter, 6);
+        sum123567 = vmls_laneq_u16(sum123567, s5, filter, 5);
+
+        uint32x4_t sum4 = vmull_laneq_u16(s4, filter, 4);
+
+        int32x4_t d0 = vaddw_s16(vreinterpretq_s32_u32(sum4),
+                                 vreinterpret_s16_u16(sum123567));
+
+        d = vqrshrun_n_s32(d0, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_u16x8(const uint16x8_t *s, uint16x8_t &d, uint16x8_t filter,
+                          uint16x8_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x8_t sum012456 = vsubq_u16(s6, s0);
+        sum012456 = vmlaq_laneq_u16(sum012456, s1, filter, 1);
+        sum012456 = vmlsq_laneq_u16(sum012456, s2, filter, 2);
+        sum012456 = vmlaq_laneq_u16(sum012456, s4, filter, 4);
+        sum012456 = vmlsq_laneq_u16(sum012456, s5, filter, 5);
+
+        uint32x4_t sum3_lo = vmull_laneq_u16(vget_low_u16(s3), filter, 3);
+        uint32x4_t sum3_hi = vmull_laneq_u16(vget_high_u16(s3), filter, 3);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum3_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum012456)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum3_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum012456)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x8_t sum07 = vaddq_u16(s0, s7);
+        uint16x8_t sum16 = vaddq_u16(s1, s6);
+        uint16x8_t sum25 = vaddq_u16(s2, s5);
+        uint16x8_t sum34 = vaddq_u16(s3, s4);
+
+        uint16x8_t sum0167 = vshlq_n_u16(sum16, 2);
+        sum0167 = vsubq_u16(sum0167, sum07);
+
+        uint32x4_t sum2345_lo = vmull_laneq_u16(vget_low_u16(sum34),
+                                                filter, 3);
+        sum2345_lo = vmlsl_laneq_u16(sum2345_lo, vget_low_u16(sum25),
+                                     filter, 2);
+
+        uint32x4_t sum2345_hi = vmull_laneq_u16(vget_high_u16(sum34),
+                                                filter, 3);
+        sum2345_hi = vmlsl_laneq_u16(sum2345_hi, vget_high_u16(sum25),
+                                     filter, 2);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum2345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0167)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum2345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0167)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x8_t sum1234567 = vsubq_u16(s1, s7);
+        sum1234567 = vmlsq_laneq_u16(sum1234567, s2, filter, 2);
+        sum1234567 = vmlaq_laneq_u16(sum1234567, s3, filter, 3);
+        sum1234567 = vmlsq_laneq_u16(sum1234567, s5, filter, 5);
+        sum1234567 = vmlaq_laneq_u16(sum1234567, s6, filter, 6);
+
+        uint32x4_t sum4_lo = vmull_laneq_u16(vget_low_u16(s4), filter, 4);
+        uint32x4_t sum4_hi = vmull_laneq_u16(vget_high_u16(s4), filter, 4);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum4_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum1234567)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum4_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum1234567)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+}
+
+#else // X265_DEPTH == 12
+template<int coeffIdx>
+void inline filter8_u16x4(const uint16x4_t *s, uint16x4_t &d,
+                          uint16x8_t filter, uint16x4_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x4_t sum0156 = vsub_u16(s6, s0);
+        sum0156 = vmla_laneq_u16(sum0156, s1, filter, 1);
+        sum0156 = vmls_laneq_u16(sum0156, s5, filter, 5);
+
+        uint32x4_t sum234 = vmull_laneq_u16(s3, filter, 3);
+        sum234 = vmlsl_laneq_u16(sum234, s2, filter, 2);
+        sum234 = vmlal_laneq_u16(sum234, s4, filter, 4);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum234),
+                                  vreinterpret_s16_u16(sum0156));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x4_t sum07 = vadd_u16(s0, s7);
+        uint16x4_t sum16 = vadd_u16(s1, s6);
+        uint16x4_t sum25 = vadd_u16(s2, s5);
+        uint16x4_t sum34 = vadd_u16(s3, s4);
+
+        uint16x4_t sum0167 = vshl_n_u16(sum16, 2);
+        sum0167 = vsub_u16(sum0167, sum07);
+
+        uint32x4_t sum2345 = vmull_laneq_u16(sum34, filter, 3);
+        sum2345 = vmlsl_laneq_u16(sum2345, sum25, filter, 2);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum2345),
+                                  vreinterpret_s16_u16(sum0167));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x4_t sum1267 = vsub_u16(s1, s7);
+        sum1267 = vmls_laneq_u16(sum1267, s2, filter, 2);
+        sum1267 = vmla_laneq_u16(sum1267, s6, filter, 6);
+
+        uint32x4_t sum345 = vmull_laneq_u16(s3, filter, 3);
+        sum345 = vmlal_laneq_u16(sum345, s4, filter, 4);
+        sum345 = vmlsl_laneq_u16(sum345, s5, filter, 5);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum345),
+                                  vreinterpret_s16_u16(sum1267));
+
+        d = vqrshrun_n_s32(sum, IF_FILTER_PREC);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_u16x8(const uint16x8_t *s, uint16x8_t &d, uint16x8_t filter,
+                          uint16x8_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x8_t sum0156 = vsubq_u16(s6, s0);
+        sum0156 = vmlaq_laneq_u16(sum0156, s1, filter, 1);
+        sum0156 = vmlsq_laneq_u16(sum0156, s5, filter, 5);
+
+        uint32x4_t sum234_lo = vmull_laneq_u16(vget_low_u16(s3), filter, 3);
+        sum234_lo = vmlsl_laneq_u16(sum234_lo, vget_low_u16(s2), filter, 2);
+        sum234_lo = vmlal_laneq_u16(sum234_lo, vget_low_u16(s4), filter, 4);
+
+        uint32x4_t sum234_hi = vmull_laneq_u16(vget_high_u16(s3), filter, 3);
+        sum234_hi = vmlsl_laneq_u16(sum234_hi, vget_high_u16(s2), filter, 2);
+        sum234_hi = vmlal_laneq_u16(sum234_hi, vget_high_u16(s4), filter, 4);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum234_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0156)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum234_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0156)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x8_t sum07 = vaddq_u16(s0, s7);
+        uint16x8_t sum16 = vaddq_u16(s1, s6);
+        uint16x8_t sum25 = vaddq_u16(s2, s5);
+        uint16x8_t sum34 = vaddq_u16(s3, s4);
+
+        uint16x8_t sum0167 = vshlq_n_u16(sum16, 2);
+        sum0167 = vsubq_u16(sum0167, sum07);
+
+        uint32x4_t sum2345_lo = vmull_laneq_u16(vget_low_u16(sum34),
+                                                filter, 3);
+        sum2345_lo = vmlsl_laneq_u16(sum2345_lo, vget_low_u16(sum25),
+                                     filter, 2);
+
+        uint32x4_t sum2345_hi = vmull_laneq_u16(vget_high_u16(sum34),
+                                                filter, 3);
+        sum2345_hi = vmlsl_laneq_u16(sum2345_hi, vget_high_u16(sum25),
+                                     filter, 2);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum2345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0167)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum2345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0167)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x8_t sum1267 = vsubq_u16(s1, s7);
+        sum1267 = vmlsq_laneq_u16(sum1267, s2, filter, 2);
+        sum1267 = vmlaq_laneq_u16(sum1267, s6, filter, 6);
+
+        uint32x4_t sum345_lo = vmull_laneq_u16(vget_low_u16(s3), filter, 3);
+        sum345_lo = vmlal_laneq_u16(sum345_lo, vget_low_u16(s4), filter, 4);
+        sum345_lo = vmlsl_laneq_u16(sum345_lo, vget_low_u16(s5), filter, 5);
+
+        uint32x4_t sum345_hi = vmull_laneq_u16(vget_high_u16(s3), filter, 3);
+        sum345_hi = vmlal_laneq_u16(sum345_hi, vget_high_u16(s4), filter, 4);
+        sum345_hi = vmlsl_laneq_u16(sum345_hi, vget_high_u16(s5), filter, 5);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum1267)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum1267)));
+
+        uint16x4_t d_lo = vqrshrun_n_s32(sum_lo, IF_FILTER_PREC);
+        uint16x4_t d_hi = vqrshrun_n_s32(sum_hi, IF_FILTER_PREC);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+}
+
+#endif // X265_DEPTH == 10
+
+template<int coeffIdx, int width, int height>
+void inline interp8_horiz_pp_neon(const pixel *src, intptr_t srcStride,
+                                  pixel *dst, intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+
+    const uint16x8_t filter =
+        vreinterpretq_u16_s16(vabsq_s16(vld1q_s16(X265_NS::g_lumaFiltercoeffIdx)));
+
+    src -= N_TAPS / 2 - 1;
+
+    for (int row = 0; row < height; row++)
+    {
+        if (width % 16 == 0)
         {
-            uint16x4_t in;
+            for (int col = 0; col < width; col += 16)
+            {
+                uint16x8_t s0N_TAPS, s1N_TAPS;
+                load_u16x8xn<8>(src + col + 0, 1, s0);
+                load_u16x8xn<8>(src + col + 8, 1, s1);
 
-#if HIGH_BIT_DEPTH
-            in = vld1_u16(src + col);
-#else
-            in = vget_low_u16(vmovl_u8(vld1_u8(src + col)));
-#endif
+                uint16x8_t d0, d1;
+                filter8_u16x8<coeffIdx>(s0, d0, filter, maxVal);
+                filter8_u16x8<coeffIdx>(s1, d1, filter, maxVal);
 
-            int16x4_t tmp = vreinterpret_s16_u16(vshl_n_u16(in, shift));
-            tmp = vsub_s16(tmp, vget_low_s16(off));
-            vst1_s16(dst + col, tmp);
+                vst1q_u16(dst + col + 0, d0);
+                vst1q_u16(dst + col + 8, d1);
+            }
         }
-        for (; col < width; col += 2)
+        else
         {
-            uint16x4_t in;
+            int col = 0;
+            for (; col + 8 <= width; col += 8)
+            {
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<8>(src + col, 1, s0);
 
-#if HIGH_BIT_DEPTH
-            in = vld1_u16(src + col);
-#else
-            in = vget_low_u16(vmovl_u8(vld1_u8(src + col)));
-#endif
+                uint16x8_t d0;
+                filter8_u16x8<coeffIdx>(s0, d0, filter, maxVal);
 
-            int16x4_t tmp = vreinterpret_s16_u16(vshl_n_u16(in, shift));
-            tmp = vsub_s16(tmp, vget_low_s16(off));
-            store_s16x2xn<1>(dst + col, dstStride, &tmp);
+                vst1q_u16(dst + col, d0);
+            }
+
+            if (width % 8 == 4)
+            {
+                uint16x4_t s0N_TAPS;
+                load_u16x4xn<8>(src + col, 1, s0);
+
+                uint16x4_t d0;
+                filter8_u16x4<coeffIdx>(s0, d0, filter, vget_low_u16(maxVal));
+
+                vst1_u16(dst + col, d0);
+            }
         }
 
         src += srcStride;
@@ -2116,73 +2922,598 @@
     }
 }
 
-#if HIGH_BIT_DEPTH
-template<int N, int width, int height>
-void interp_horiz_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx)
+#if X265_DEPTH == 10
+template<int coeff4>
+void inline filter4_ps_u16x4(const uint16x4_t *s, const uint16x4_t f,
+                             const uint16x8_t offset, int16x4_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x4_t sum03 = vadd_u16(s0, s3);
+        uint16x4_t sum12 = vadd_u16(s1, s2);
+
+        int16x4_t sum =
+            vreinterpret_s16_u16(vmla_n_u16(vget_low_u16(offset), sum12, 9));
+        d = vsub_s16(sum, vreinterpret_s16_u16(sum03));
+    }
+    else
+    {
+        uint16x4_t sum = vmls_lane_u16(vget_low_u16(offset), s0, f, 0);
+        sum = vmla_lane_u16(sum, s1, f, 1);
+        sum = vmla_lane_u16(sum, s2, f, 2);
+        sum = vmls_lane_u16(sum, s3, f, 3);
+
+        // We halved filter values so -1 from right shift.
+        d = vshr_n_s16(vreinterpret_s16_u16(sum), SHIFT_INTERP_PS - 1);
+    }
+}
+
+template<bool coeff4>
+void inline filter4_ps_u16x8(const uint16x8_t *s, const uint16x4_t f,
+                             const uint16x8_t offset, int16x8_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x8_t sum03 = vaddq_u16(s0, s3);
+        uint16x8_t sum12 = vaddq_u16(s1, s2);
+
+        int16x8_t sum =
+            vreinterpretq_s16_u16(vmlaq_n_u16(offset, sum12, 9));
+        d = vsubq_s16(sum, vreinterpretq_s16_u16(sum03));
+    }
+    else
+    {
+        uint16x8_t sum = vmlsq_lane_u16(offset, s0, f, 0);
+        sum = vmlaq_lane_u16(sum, s1, f, 1);
+        sum = vmlaq_lane_u16(sum, s2, f, 2);
+        sum = vmlsq_lane_u16(sum, s3, f, 3);
+
+        // We halved filter values so -1 from right shift.
+        d = vshrq_n_s16(vreinterpretq_s16_u16(sum), SHIFT_INTERP_PS - 1);
+    }
+}
+
+#else // X265_DEPTH == 12
+template<int coeff4>
+void inline filter4_ps_u16x4(const uint16x4_t *s, const uint16x4_t f,
+                             const uint32x4_t offset, int16x4_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x4_t sum03 = vadd_u16(s0, s3);
+        uint16x4_t sum12 = vadd_u16(s1, s2);
+
+        int32x4_t sum = vreinterpretq_s32_u32(vmlal_n_u16(offset, sum12, 9));
+        sum = vsubw_s16(sum, vreinterpret_s16_u16(sum03));
+
+        // We divided filter values by 4 so -2 from right shift.
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS - 2);
+    }
+    else
+    {
+        uint32x4_t sum = vmlsl_lane_u16(offset, s0, f, 0);
+        sum = vmlal_lane_u16(sum, s1, f, 1);
+        sum = vmlal_lane_u16(sum, s2, f, 2);
+        sum = vmlsl_lane_u16(sum, s3, f, 3);
+
+        d = vshrn_n_s32(vreinterpretq_s32_u32(sum), SHIFT_INTERP_PS);
+    }
+}
+
+template<bool coeff4>
+void inline filter4_ps_u16x8(const uint16x8_t *s, const uint16x4_t f,
+                             const uint32x4_t offset, int16x8_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        uint16x8_t sum03 = vaddq_u16(s0, s3);
+        uint16x8_t sum12 = vaddq_u16(s1, s2);
+
+        int32x4_t sum_lo = vreinterpretq_s32_u32(
+            vmlal_n_u16(offset, vget_low_u16(sum12), 9));
+        int32x4_t sum_hi = vreinterpretq_s32_u32(
+            vmlal_n_u16(offset, vget_high_u16(sum12), 9));
+        sum_lo = vsubw_s16(sum_lo, vreinterpret_s16_u16(vget_low_u16(sum03)));
+        sum_hi = vsubw_s16(sum_hi, vreinterpret_s16_u16(vget_high_u16(sum03)));
+
+        // We divided filter values by 4 so -2 from right shift.
+        int16x4_t d0 = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS - 2);
+        int16x4_t d1 = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS - 2);
+        d = vcombine_s16(d0, d1);
+    }
+    else
+    {
+        uint32x4_t sum_lo = vmlsl_lane_u16(offset, vget_low_u16(s0), f, 0);
+        sum_lo = vmlal_lane_u16(sum_lo, vget_low_u16(s1), f, 1);
+        sum_lo = vmlal_lane_u16(sum_lo, vget_low_u16(s2), f, 2);
+        sum_lo = vmlsl_lane_u16(sum_lo, vget_low_u16(s3), f, 3);
+
+        uint32x4_t sum_hi = vmlsl_lane_u16(offset, vget_high_u16(s0), f, 0);
+        sum_hi = vmlal_lane_u16(sum_hi, vget_high_u16(s1), f, 1);
+        sum_hi = vmlal_lane_u16(sum_hi, vget_high_u16(s2), f, 2);
+        sum_hi = vmlsl_lane_u16(sum_hi, vget_high_u16(s3), f, 3);
+
+        int16x4_t d0 = vshrn_n_s32(vreinterpretq_s32_u32(sum_lo),
+                                   SHIFT_INTERP_PS);
+        int16x4_t d1 = vshrn_n_s32(vreinterpretq_s32_u32(sum_hi),
+                                   SHIFT_INTERP_PS);
+        d = vcombine_s16(d0, d1);
+    }
+}
+
+#endif // X265_DEPTH == 10
+
+template<int coeff4, int width, int height>
+void interp4_horiz_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride, int coeffIdx, int isRowExt)
 {
-    const int16_t *coeff = (N == 4) ? g_chromaFiltercoeffIdx : g_lumaFiltercoeffIdx;
-    int headRoom = IF_FILTER_PREC;
-    int offset = (1 << (headRoom - 1));
-    uint16_t maxVal = (1 << X265_DEPTH) - 1;
-    int cStride = 1;
+    const int N_TAPS = 4;
+    int blkheight = height;
+    uint16x4_t filter = vreinterpret_u16_s16(
+        vabs_s16(vld1_s16(X265_NS::g_chromaFiltercoeffIdx)));
+    uint32_t offset_u32;
 
-    src -= (N / 2 - 1) * cStride;
-    int16x8_t vc = vld1q_s16(coeff);
-    int16x4_t low_vc = vget_low_s16(vc);
-    int16x4_t high_vc = vget_high_s16(vc);
+    if (coeff4)
+    {
+        // The -2 is needed because we will divide the filter values by 4.
+        offset_u32 = (unsigned)-IF_INTERNAL_OFFS << (SHIFT_INTERP_PS - 2);
+    }
+    else
+    {
+        offset_u32 = (unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS;
+    }
+#if X265_DEPTH == 10
+    if (!coeff4)
+    {
+        // All filter values are even, halve them to avoid needing to widen to
+        // 32-bit elements in filter kernels.
+        filter = vshr_n_u16(filter, 1);
+        offset_u32 >>= 1;
+    }
+
+    const uint16x8_t offset = vdupq_n_u16((uint16_t)offset_u32);
+#else
+    const uint32x4_t offset = vdupq_n_u32(offset_u32);
+#endif // X265_DEPTH == 10
+
+    if (isRowExt)
+    {
+        src -= (N_TAPS / 2 - 1) * srcStride;
+        blkheight += N_TAPS - 1;
+    }
 
-    const int32x4_t voffset = vdupq_n_s32(offset);
-    const int32x4_t vhr = vdupq_n_s32(-headRoom);
+    src -= N_TAPS / 2 - 1;
 
-    int row, col;
-    for (row = 0; row < height; row++)
+    for (int row = 0; row < blkheight; row++)
     {
-        for (col = 0; col < width; col += 8)
+        if (width % 16 == 0)
+        {
+            for (int col = 0; col < width; col += 16)
+            {
+                uint16x8_t s0N_TAPS, s1N_TAPS;
+                load_u16x8xn<4>(src + col + 0, 1, s0);
+                load_u16x8xn<4>(src + col + 8, 1, s1);
+
+                int16x8_t d0, d1;
+                filter4_ps_u16x8<coeff4>(s0, filter, offset, d0);
+                filter4_ps_u16x8<coeff4>(s1, filter, offset, d1);
+
+                vst1q_s16(dst + col + 0, d0);
+                vst1q_s16(dst + col + 8, d1);
+            }
+        }
+        else
         {
-            int32x4_t vsum1, vsum2;
+            int col = 0;
+            for (; col + 8 <= width; col += 8)
+            {
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<4>(src + col, 1, s0);
+
+                int16x8_t d0;
+                filter4_ps_u16x8<coeff4>(s0, filter, offset, d0);
 
-            int16x8_t inputN;
+                vst1q_s16(dst + col, d0);
+            }
 
-            for (int i = 0; i < N; i++)
+            if (width == 6)
             {
-                inputi = vreinterpretq_s16_u16(vld1q_u16(src + col + i));
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<4>(src, 1, s0);
+
+                int16x8_t d0;
+                filter4_ps_u16x8<coeff4>(s0, filter, offset, d0);
+
+                store_s16x6xn<1>(dst, dstStride, &d0);
+            }
+            else if (width % 8 != 0)
+            {
+                uint16x4_t s0N_TAPS;
+                load_u16x4xn<4>(src + col, 1, s0);
+
+                int16x4_t d0;
+                filter4_ps_u16x4<coeff4>(s0, filter, offset, d0);
+
+                if (width == 2)
+                {
+                    store_s16x2xn<1>(dst + col, dstStride, &d0);
+                }
+                else
+                {
+                    vst1_s16(dst + col, d0);
+                }
             }
-            vsum1 = voffset;
-            vsum2 = voffset;
+        }
 
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input0), low_vc, 0);
-            vsum2 = vmlal_high_lane_s16(vsum2, input0, low_vc, 0);
+        src += srcStride;
+        dst += dstStride;
+    }
+}
+
+#if X265_DEPTH == 10
+template<int coeffIdx>
+void inline filter8_ps_u16x4(const uint16x4_t *s, int16x4_t &d,
+                             uint32x4_t offset, uint16x8_t filter)
+{
+    uint16x4_t offset_u16 = vdup_n_u16((uint16_t)vgetq_lane_u32(offset, 0));
+
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x4_t sum012456 = vsub_u16(s6, s0);
+        sum012456 = vmla_laneq_u16(sum012456, s1, filter, 1);
+        sum012456 = vmls_laneq_u16(sum012456, s2, filter, 2);
+        sum012456 = vmla_laneq_u16(sum012456, s4, filter, 4);
+        sum012456 = vmls_laneq_u16(sum012456, s5, filter, 5);
+
+        uint16x4_t sum3 =
+            vmla_laneq_u16(offset_u16, s3, filter, 3);
+
+        int32x4_t sum = vaddl_s16(vreinterpret_s16_u16(sum3),
+                                  vreinterpret_s16_u16(sum012456));
+
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x4_t sum07 = vadd_u16(s0, s7);
+        uint16x4_t sum16 = vadd_u16(s1, s6);
+        uint16x4_t sum25 = vadd_u16(s2, s5);
+        uint16x4_t sum34 = vadd_u16(s3, s4);
+
+        uint16x4_t sum0167 = vshl_n_u16(sum16, 2);
+        sum0167 = vsub_u16(sum0167, sum07);
+
+        uint32x4_t sum2345 = vmlal_laneq_u16(offset, sum34, filter, 3);
+        sum2345 = vmlsl_laneq_u16(sum2345, sum25, filter, 2);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum2345),
+                                  vreinterpret_s16_u16(sum0167));
 
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input1), low_vc, 1);
-            vsum2 = vmlal_high_lane_s16(vsum2, input1, low_vc, 1);
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x4_t sum123567 = vsub_u16(s1, s7);
+        sum123567 = vmls_laneq_u16(sum123567, s2, filter, 2);
+        sum123567 = vmla_laneq_u16(sum123567, s3, filter, 3);
+        sum123567 = vmla_laneq_u16(sum123567, s6, filter, 6);
+        sum123567 = vmls_laneq_u16(sum123567, s5, filter, 5);
 
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input2), low_vc, 2);
-            vsum2 = vmlal_high_lane_s16(vsum2, input2, low_vc, 2);
+        uint16x4_t sum4 =
+            vmla_laneq_u16(offset_u16, s4, filter, 4);
 
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input3), low_vc, 3);
-            vsum2 = vmlal_high_lane_s16(vsum2, input3, low_vc, 3);
+        int32x4_t sum = vaddl_s16(vreinterpret_s16_u16(sum4),
+                                  vreinterpret_s16_u16(sum123567));
+
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+}
 
-            if (N == 8)
+template<int coeffIdx>
+void inline filter8_ps_u16x8(const uint16x8_t *s, int16x8_t &d,
+                             uint32x4_t offset, uint16x8_t filter)
+{
+    uint16x8_t offset_u16 = vdupq_n_u16((uint16_t)vgetq_lane_u32(offset, 0));
+
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x8_t sum012456 = vsubq_u16(s6, s0);
+        sum012456 = vmlaq_laneq_u16(sum012456, s1, filter, 1);
+        sum012456 = vmlsq_laneq_u16(sum012456, s2, filter, 2);
+        sum012456 = vmlaq_laneq_u16(sum012456, s4, filter, 4);
+        sum012456 = vmlsq_laneq_u16(sum012456, s5, filter, 5);
+
+        uint16x8_t sum3 =
+            vmlaq_laneq_u16(offset_u16, s3, filter, 3);
+
+        int32x4_t sum_lo = vaddl_s16(vget_low_s16(vreinterpretq_s16_u16(sum3)),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum012456)));
+        int32x4_t sum_hi = vaddl_s16(vget_high_s16(vreinterpretq_s16_u16(sum3)),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum012456)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+        d = vcombine_s16(d_lo, d_hi);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x8_t sum07 = vaddq_u16(s0, s7);
+        uint16x8_t sum16 = vaddq_u16(s1, s6);
+        uint16x8_t sum25 = vaddq_u16(s2, s5);
+        uint16x8_t sum34 = vaddq_u16(s3, s4);
+
+        uint16x8_t sum0167 = vshlq_n_u16(sum16, 2);
+        sum0167 = vsubq_u16(sum0167, sum07);
+
+        uint32x4_t sum2345_lo = vmlal_laneq_u16(offset, vget_low_u16(sum34),
+                                                filter, 3);
+        sum2345_lo = vmlsl_laneq_u16(sum2345_lo, vget_low_u16(sum25),
+                                     filter, 2);
+
+        uint32x4_t sum2345_hi = vmlal_laneq_u16(offset, vget_high_u16(sum34),
+                                                filter, 3);
+        sum2345_hi = vmlsl_laneq_u16(sum2345_hi, vget_high_u16(sum25),
+                                     filter, 2);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum2345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0167)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum2345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0167)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+        d = vcombine_s16(d_lo, d_hi);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x8_t sum123567 = vsubq_u16(s1, s7);
+        sum123567 = vmlsq_laneq_u16(sum123567, s2, filter, 2);
+        sum123567 = vmlaq_laneq_u16(sum123567, s3, filter, 3);
+        sum123567 = vmlaq_laneq_u16(sum123567, s6, filter, 6);
+        sum123567 = vmlsq_laneq_u16(sum123567, s5, filter, 5);
+
+        uint16x8_t sum4 =
+            vmlaq_laneq_u16(offset_u16, s4, filter, 4);
+
+        int32x4_t sum_lo = vaddl_s16(vget_low_s16(vreinterpretq_s16_u16(sum4)),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum123567)));
+        int32x4_t sum_hi = vaddl_s16(vget_high_s16(vreinterpretq_s16_u16(sum4)),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum123567)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+        d = vcombine_s16(d_lo, d_hi);
+    }
+}
+
+#else // X265_DEPTH == 12
+template<int coeffIdx>
+void inline filter8_ps_u16x4(const uint16x4_t *s, int16x4_t &d,
+                             uint32x4_t offset, uint16x8_t filter)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x4_t sum0156 = vsub_u16(s6, s0);
+        sum0156 = vmla_laneq_u16(sum0156, s1, filter, 1);
+        sum0156 = vmls_laneq_u16(sum0156, s5, filter, 5);
+
+        uint32x4_t sum234 = vmlal_laneq_u16(offset, s3, filter, 3);
+        sum234 = vmlsl_laneq_u16(sum234, s2, filter, 2);
+        sum234 = vmlal_laneq_u16(sum234, s4, filter, 4);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum234),
+                                  vreinterpret_s16_u16(sum0156));
+
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x4_t sum07 = vadd_u16(s0, s7);
+        uint16x4_t sum16 = vadd_u16(s1, s6);
+        uint16x4_t sum25 = vadd_u16(s2, s5);
+        uint16x4_t sum34 = vadd_u16(s3, s4);
+
+        uint16x4_t sum0167 = vshl_n_u16(sum16, 2);
+        sum0167 = vsub_u16(sum0167, sum07);
+
+        uint32x4_t sum2345 = vmlal_laneq_u16(offset, sum34, filter, 3);
+        sum2345 = vmlsl_laneq_u16(sum2345, sum25, filter, 2);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum2345),
+                                  vreinterpret_s16_u16(sum0167));
+
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x4_t sum1267 = vsub_u16(s1, s7);
+        sum1267 = vmls_laneq_u16(sum1267, s2, filter, 2);
+        sum1267 = vmla_laneq_u16(sum1267, s6, filter, 6);
+
+        uint32x4_t sum345 = vmlal_laneq_u16(offset, s3, filter, 3);
+        sum345 = vmlal_laneq_u16(sum345, s4, filter, 4);
+        sum345 = vmlsl_laneq_u16(sum345, s5, filter, 5);
+
+        int32x4_t sum = vaddw_s16(vreinterpretq_s32_u32(sum345),
+                                  vreinterpret_s16_u16(sum1267));
+
+        d = vshrn_n_s32(sum, SHIFT_INTERP_PS);
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_ps_u16x8(const uint16x8_t *s, int16x8_t &d,
+                             uint32x4_t offset, uint16x8_t filter)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        uint16x8_t sum0156 = vsubq_u16(s6, s0);
+        sum0156 = vmlaq_laneq_u16(sum0156, s1, filter, 1);
+        sum0156 = vmlsq_laneq_u16(sum0156, s5, filter, 5);
+
+        uint32x4_t sum234_lo = vmlal_laneq_u16(offset, vget_low_u16(s3), filter, 3);
+        sum234_lo = vmlsl_laneq_u16(sum234_lo, vget_low_u16(s2), filter, 2);
+        sum234_lo = vmlal_laneq_u16(sum234_lo, vget_low_u16(s4), filter, 4);
+
+        uint32x4_t sum234_hi = vmlal_laneq_u16(offset, vget_high_u16(s3), filter, 3);
+        sum234_hi = vmlsl_laneq_u16(sum234_hi, vget_high_u16(s2), filter, 2);
+        sum234_hi = vmlal_laneq_u16(sum234_hi, vget_high_u16(s4), filter, 4);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum234_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0156)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum234_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0156)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+        d = vcombine_s16(d_lo, d_hi);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        uint16x8_t sum07 = vaddq_u16(s0, s7);
+        uint16x8_t sum16 = vaddq_u16(s1, s6);
+        uint16x8_t sum25 = vaddq_u16(s2, s5);
+        uint16x8_t sum34 = vaddq_u16(s3, s4);
+
+        uint16x8_t sum0167 = vshlq_n_u16(sum16, 2);
+        sum0167 = vsubq_u16(sum0167, sum07);
+
+        uint32x4_t sum2345_lo = vmlal_laneq_u16(offset, vget_low_u16(sum34),
+                                                filter, 3);
+        sum2345_lo = vmlsl_laneq_u16(sum2345_lo, vget_low_u16(sum25),
+                                     filter, 2);
+
+        uint32x4_t sum2345_hi = vmlal_laneq_u16(offset, vget_high_u16(sum34),
+                                                filter, 3);
+        sum2345_hi = vmlsl_laneq_u16(sum2345_hi, vget_high_u16(sum25),
+                                     filter, 2);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum2345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum0167)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum2345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum0167)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+        d = vcombine_s16(d_lo, d_hi);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        uint16x8_t sum1267 = vsubq_u16(s1, s7);
+        sum1267 = vmlsq_laneq_u16(sum1267, s2, filter, 2);
+        sum1267 = vmlaq_laneq_u16(sum1267, s6, filter, 6);
+
+        uint32x4_t sum345_lo = vmlal_laneq_u16(offset, vget_low_u16(s3), filter, 3);
+        sum345_lo = vmlal_laneq_u16(sum345_lo, vget_low_u16(s4), filter, 4);
+        sum345_lo = vmlsl_laneq_u16(sum345_lo, vget_low_u16(s5), filter, 5);
+
+        uint32x4_t sum345_hi = vmlal_laneq_u16(offset, vget_high_u16(s3), filter, 3);
+        sum345_hi = vmlal_laneq_u16(sum345_hi, vget_high_u16(s4), filter, 4);
+        sum345_hi = vmlsl_laneq_u16(sum345_hi, vget_high_u16(s5), filter, 5);
+
+        int32x4_t sum_lo = vaddw_s16(vreinterpretq_s32_u32(sum345_lo),
+                                     vget_low_s16(vreinterpretq_s16_u16(sum1267)));
+        int32x4_t sum_hi = vaddw_s16(vreinterpretq_s32_u32(sum345_hi),
+                                     vget_high_s16(vreinterpretq_s16_u16(sum1267)));
+
+        int16x4_t d_lo = vshrn_n_s32(sum_lo, SHIFT_INTERP_PS);
+        int16x4_t d_hi = vshrn_n_s32(sum_hi, SHIFT_INTERP_PS);
+
+        d = vcombine_s16(d_lo, d_hi);
+    }
+}
+
+#endif // X265_DEPTH == 10
+
+template<int coeffIdx, int width, int height>
+void interp8_horiz_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride, int isRowExt)
+{
+    const int N_TAPS = 8;
+    int blkheight = height;
+    const uint16x8_t filter =
+        vreinterpretq_u16_s16(vabsq_s16(vld1q_s16(X265_NS::g_lumaFiltercoeffIdx)));
+    uint32x4_t offset =
+        vdupq_n_u32((unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS);
+
+    if (isRowExt)
+    {
+        src -= (N_TAPS / 2 - 1) * srcStride;
+        blkheight += N_TAPS - 1;
+    }
+
+    src -= N_TAPS / 2 - 1;
+
+    for (int row = 0; row < blkheight; row++)
+    {
+        if (width % 16 == 0)
+        {
+            for (int col = 0; col < width; col += 16)
+            {
+                uint16x8_t s0N_TAPS, s1N_TAPS;
+                load_u16x8xn<8>(src + col + 0, 1, s0);
+                load_u16x8xn<8>(src + col + 8, 1, s1);
+
+                int16x8_t d0, d1;
+                filter8_ps_u16x8<coeffIdx>(s0, d0, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(s1, d1, offset, filter);
+
+                vst1q_s16(dst + col + 0, d0);
+                vst1q_s16(dst + col + 8, d1);
+            }
+        }
+        else
+        {
+            int col = 0;
+            for (; col + 8 <= width; col += 8)
             {
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input4), high_vc, 0);
-                vsum2 = vmlal_high_lane_s16(vsum2, input4, high_vc, 0);
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input5), high_vc, 1);
-                vsum2 = vmlal_high_lane_s16(vsum2, input5, high_vc, 1);
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input6), high_vc, 2);
-                vsum2 = vmlal_high_lane_s16(vsum2, input6, high_vc, 2);
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input7), high_vc, 3);
-                vsum2 = vmlal_high_lane_s16(vsum2, input7, high_vc, 3);
+                uint16x8_t s0N_TAPS;
+                load_u16x8xn<8>(src + col, 1, s0);
+
+                int16x8_t d0;
+                filter8_ps_u16x8<coeffIdx>(s0, d0, offset, filter);
 
+                vst1q_s16(dst + col, d0);
             }
 
-            vsum1 = vshlq_s32(vsum1, vhr);
-            vsum2 = vshlq_s32(vsum2, vhr);
+            if (width % 8 == 4)
+            {
+                uint16x4_t s0N_TAPS;
+                load_u16x4xn<8>(src + col, 1, s0);
+
+                int16x4_t d0;
+                filter8_ps_u16x4<coeffIdx>(s0, d0, offset, filter);
 
-            int16x8_t vsum = vuzp1q_s16(vreinterpretq_s16_s32(vsum1),
-                                        vreinterpretq_s16_s32(vsum2));
-            vsum = vminq_s16(vsum, vdupq_n_s16(maxVal));
-            vsum = vmaxq_s16(vsum, vdupq_n_s16(0));
-            vst1q_u16(dst + col, vreinterpretq_u16_s16(vsum));
+                vst1_s16(dst + col, d0);
+            }
         }
 
         src += srcStride;
@@ -2190,125 +3521,1551 @@
     }
 }
 
-#else // HIGH_BIT_DEPTH
-template<int N, int width, int height>
-void interp_horiz_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
-                          intptr_t dstStride, int coeffIdx)
+template<bool coeff4, int width, int height>
+void inline interp4_vert_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                                 intptr_t dstStride, const int16_t coeffIdx)
 {
-    if (N == 8)
+    const int N_TAPS = 4;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    uint16x4_t filter = vreinterpret_u16_s16(
+        vabs_s16(vld1_s16(X265_NS::g_chromaFiltercoeffIdx)));
+
+    uint16_t offset_u16;
+
+    // A shim of 1 << (IF_FILTER_PREC - 1) enables us to use non-rounding
+    // shifts - which are generally faster than rounding shifts on modern CPUs.
+    if (coeff4)
     {
-        switch (coeffIdx)
+        // The outermost -2 is needed because we will divide the filter values by 4.
+        offset_u16 = 1 << (IF_FILTER_PREC - 1 - 2);
+    }
+    else
+    {
+        offset_u16 = 1 << (IF_FILTER_PREC - 1);
+    }
+
+#if X265_DEPTH == 10
+    if (!coeff4)
+    {
+        // All filter values are even, halve them to avoid needing to widen to
+        // 32-bit elements in filter kernels.
+        filter = vshr_n_u16(filter, 1);
+        offset_u16 >>= 1;
+    }
+
+    const uint16x8_t offset = vdupq_n_u16(offset_u16);
+#else
+    const uint32x4_t offset = vdupq_n_u32(offset_u16);
+#endif // X265_DEPTH == 10
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        if (width == 12 || width == 6)
         {
-        case 1:
-            return interp8_horiz_pp_neon<1, width, height>(src, srcStride, dst,
-                                                           dstStride);
-        case 2:
-            return interp8_horiz_pp_neon<2, width, height>(src, srcStride, dst,
-                                                           dstStride);
-        case 3:
-            return interp8_horiz_pp_neon<3, width, height>(src, srcStride, dst,
-                                                           dstStride);
+            const int n_store = width == 12 ? 8 : 6;
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in07;
+            load_u16x8xn<3>(s, srcStride, in0);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in0 + 3);
+
+                uint16x8_t res4;
+                filter4_u16x8<coeff4>(in0 + 0, filter, offset, maxVal, res0);
+                filter4_u16x8<coeff4>(in0 + 1, filter, offset, maxVal, res1);
+                filter4_u16x8<coeff4>(in0 + 2, filter, offset, maxVal, res2);
+                filter4_u16x8<coeff4>(in0 + 3, filter, offset, maxVal, res3);
+
+                store_u16xnxm<n_store, 4>(d, dstStride, res);
+
+                in00 = in04;
+                in01 = in05;
+                in02 = in06;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (width == 6)
+            {
+                return;
+            }
+
+            src += 8;
+            dst += 8;
+        }
+
+        const int n_store = width > 4 ? 4 : width;
+
+        uint16x4_t in17;
+        load_u16x4xn<3>(src, srcStride, in1);
+        src += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_u16x4xn<4>(src, srcStride, in1 + 3);
+
+            uint16x4_t res4;
+            filter4_u16x4<coeff4>(in1 + 0, filter, offset, vget_low_u16(maxVal), res0);
+            filter4_u16x4<coeff4>(in1 + 1, filter, offset, vget_low_u16(maxVal), res1);
+            filter4_u16x4<coeff4>(in1 + 2, filter, offset, vget_low_u16(maxVal), res2);
+            filter4_u16x4<coeff4>(in1 + 3, filter, offset, vget_low_u16(maxVal), res3);
+
+            store_u16xnxm<n_store, 4>(dst, dstStride, res);
+
+            in10 = in14;
+            in11 = in15;
+            in12 = in16;
+
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_u16x4xn<2>(src, srcStride, in1 + 3);
+
+            uint16x4_t res2;
+            filter4_u16x4<coeff4>(in1 + 0, filter, offset, vget_low_u16(maxVal), res0);
+            filter4_u16x4<coeff4>(in1 + 1, filter, offset, vget_low_u16(maxVal), res1);
+
+            store_u16xnxm<n_store, 2>(dst, dstStride, res);
         }
     }
     else
     {
-        switch (coeffIdx)
+        for (int col = 0; col < width; col += 8)
         {
-        case 4:
-            return interp4_horiz_pp_neon<true, width, height>(src, srcStride,
-                                                              dst, dstStride,
-                                                              coeffIdx);
-        default:
-            return interp4_horiz_pp_neon<false, width, height>(src, srcStride,
-                                                               dst, dstStride,
-                                                               coeffIdx);
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in7;
+            load_u16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in + 3);
+
+                uint16x8_t res4;
+                filter4_u16x8<coeff4>(in + 0, filter, offset, maxVal, res0);
+                filter4_u16x8<coeff4>(in + 1, filter, offset, maxVal, res1);
+                filter4_u16x8<coeff4>(in + 2, filter, offset, maxVal, res2);
+                filter4_u16x8<coeff4>(in + 3, filter, offset, maxVal, res3);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (height & 2)
+            {
+                load_u16x8xn<2>(s, srcStride, in + 3);
+
+                uint16x8_t res2;
+                filter4_u16x8<coeff4>(in + 0, filter, offset, maxVal, res0);
+                filter4_u16x8<coeff4>(in + 1, filter,  offset, maxVal, res1);
+
+                store_u16x8xn<2>(d, dstStride, res);
+            }
+
+            src += 8;
+            dst += 8;
         }
     }
 }
 
-#endif // HIGH_BIT_DEPTH
+template<int coeffIdx, int width, int height>
+void inline interp8_vert_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                                 intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    const uint16x8_t filter =
+        vreinterpretq_u16_s16(vabsq_s16(vld1q_s16(X265_NS::g_lumaFiltercoeffIdx)));
 
-#if HIGH_BIT_DEPTH
+    src -= (N_TAPS / 2 - 1) * srcStride;
 
-template<int N, int width, int height>
-void interp_horiz_ps_neon(const uint16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx,
-                          int isRowExt)
+    if (width % 8 != 0)
+    {
+        const uint16_t *s = src;
+        uint16_t *d = dst;
+
+        if (width == 12)
+        {
+            uint16x8_t in11;
+            load_u16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in + 7);
+
+                uint16x8_t res4;
+                filter8_u16x8<coeffIdx>(in + 0, res0, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 1, res1, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 2, res2, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 3, res3, filter, maxVal);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            s = src + 8;
+            d = dst + 8;
+        }
+
+        uint16x4_t in11;
+        load_u16x4xn<7>(s, srcStride, in);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_u16x4xn<4>(s, srcStride, in + 7);
+
+            uint16x4_t res4;
+            filter8_u16x4<coeffIdx>(in + 0, res0, filter, vget_low_u16(maxVal));
+            filter8_u16x4<coeffIdx>(in + 1, res1, filter, vget_low_u16(maxVal));
+            filter8_u16x4<coeffIdx>(in + 2, res2, filter, vget_low_u16(maxVal));
+            filter8_u16x4<coeffIdx>(in + 3, res3, filter, vget_low_u16(maxVal));
+
+            store_u16x4xn<4>(d, dstStride, res);
+
+            in0 = in4;
+            in1 = in5;
+            in2 = in6;
+            in3 = in7;
+            in4 = in8;
+            in5 = in9;
+            in6 = in10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+    }
+    else if (width % 16 != 0)
+    {
+        for (int col = 0; col < width; col += 8)
+        {
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in11;
+            load_u16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in + 7);
+
+                uint16x8_t res4;
+                filter8_u16x8<coeffIdx>(in + 0, res0, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 1, res1, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 2, res2, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in + 3, res3, filter, maxVal);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            src += 8;
+            dst += 8;
+        }
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 16)
+        {
+            const uint16_t *s = src;
+            uint16_t *d = dst;
+
+            uint16x8_t in011, in111;
+            load_u16x8xn<7>(s + 0, srcStride, in0);
+            load_u16x8xn<7>(s + 8, srcStride, in1);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_u16x8xn<4>(s + 0, srcStride, in0 + 7);
+                load_u16x8xn<4>(s + 8, srcStride, in1 + 7);
+
+                uint16x8_t res04, res14;
+                filter8_u16x8<coeffIdx>(in0 + 0, res00, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in0 + 1, res01, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in0 + 2, res02, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in0 + 3, res03, filter, maxVal);
+
+                filter8_u16x8<coeffIdx>(in1 + 0, res10, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in1 + 1, res11, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in1 + 2, res12, filter, maxVal);
+                filter8_u16x8<coeffIdx>(in1 + 3, res13, filter, maxVal);
+
+                store_u16x8xn<4>(d + 0, dstStride, res0);
+                store_u16x8xn<4>(d + 8, dstStride, res1);
+
+                in00 = in04;
+                in01 = in05;
+                in02 = in06;
+                in03 = in07;
+                in04 = in08;
+                in05 = in09;
+                in06 = in010;
+
+                in10 = in14;
+                in11 = in15;
+                in12 = in16;
+                in13 = in17;
+                in14 = in18;
+                in15 = in19;
+                in16 = in110;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            src += 16;
+            dst += 16;
+        }
+    }
+}
+
+template<bool coeff4, int width, int height>
+void inline interp4_vert_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                                 intptr_t dstStride, const int16_t coeffIdx)
 {
-    const int16_t *coeff = (N == 4) ? g_chromaFiltercoeffIdx : g_lumaFiltercoeffIdx;
-    const int offset = (unsigned) - IF_INTERNAL_OFFS << SHIFT_INTERP_PS;
+    const int N_TAPS = 4;
+    uint16x4_t filter = vreinterpret_u16_s16(
+        vabs_s16(vld1_s16(X265_NS::g_chromaFiltercoeffIdx)));
+    uint32_t offset_u32;
 
-    int blkheight = height;
-    src -= N / 2 - 1;
+    if (coeff4)
+    {
+        // The -2 is needed because we will divide the filter values by 4.
+        offset_u32 = (unsigned)-IF_INTERNAL_OFFS << (SHIFT_INTERP_PS - 2);
+    }
+    else
+    {
+        offset_u32 = (unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS;
+    }
+#if X265_DEPTH == 10
+    if (!coeff4)
+    {
+        // All filter values are even, halve them to avoid needing to widen to
+        // 32-bit elements in filter kernels.
+        filter = vshr_n_u16(filter, 1);
+        offset_u32 >>= 1;
+    }
 
-    if (isRowExt)
+    const uint16x8_t offset = vdupq_n_u16((uint16_t)offset_u32);
+#else
+    const uint32x4_t offset = vdupq_n_u32(offset_u32);
+#endif // X265_DEPTH == 10
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        if (width == 12 || width == 6)
+        {
+            const int n_store = width == 12 ? 8 : 6;
+            const uint16_t *s = src;
+            int16_t *d = dst;
+
+            uint16x8_t in07;
+            load_u16x8xn<3>(s, srcStride, in0);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in0 + 3);
+
+                int16x8_t res4;
+                filter4_ps_u16x8<coeff4>(in0 + 0, filter, offset, res0);
+                filter4_ps_u16x8<coeff4>(in0 + 1, filter, offset, res1);
+                filter4_ps_u16x8<coeff4>(in0 + 2, filter, offset, res2);
+                filter4_ps_u16x8<coeff4>(in0 + 3, filter, offset, res3);
+
+                store_s16xnxm<n_store, 4>(res, d, dstStride);
+
+                in00 = in04;
+                in01 = in05;
+                in02 = in06;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (width == 6)
+            {
+                return;
+            }
+
+            src += 8;
+            dst += 8;
+        }
+
+        const int n_store = width > 4 ? 4 : width;
+
+        uint16x4_t in17;
+        load_u16x4xn<3>(src, srcStride, in1);
+        src += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_u16x4xn<4>(src, srcStride, in1 + 3);
+
+            int16x4_t res4;
+            filter4_ps_u16x4<coeff4>(in1 + 0, filter, offset, res0);
+            filter4_ps_u16x4<coeff4>(in1 + 1, filter, offset, res1);
+            filter4_ps_u16x4<coeff4>(in1 + 2, filter, offset, res2);
+            filter4_ps_u16x4<coeff4>(in1 + 3, filter, offset, res3);
+
+            store_s16xnxm<n_store, 4>(res, dst, dstStride);
+
+            in10 = in14;
+            in11 = in15;
+            in12 = in16;
+
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_u16x4xn<2>(src, srcStride, in1 + 3);
+
+            int16x4_t res2;
+            filter4_ps_u16x4<coeff4>(in1 + 0, filter, offset, res0);
+            filter4_ps_u16x4<coeff4>(in1 + 1, filter, offset, res1);
+
+            store_s16xnxm<n_store, 2>(res, dst, dstStride);
+        }
+    }
+    else
     {
-        src -= (N / 2 - 1) * srcStride;
-        blkheight += N - 1;
+        for (int col = 0; col < width; col += 8)
+        {
+            const uint16_t *s = src;
+            int16_t *d = dst;
+
+            uint16x8_t in7;
+            load_u16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_u16x8xn<4>(s, srcStride, in + 3);
+
+                int16x8_t res4;
+                filter4_ps_u16x8<coeff4>(in + 0, filter, offset, res0);
+                filter4_ps_u16x8<coeff4>(in + 1, filter, offset, res1);
+                filter4_ps_u16x8<coeff4>(in + 2, filter, offset, res2);
+                filter4_ps_u16x8<coeff4>(in + 3, filter, offset, res3);
+
+                store_s16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (height & 2)
+            {
+                load_u16x8xn<2>(s, srcStride, in + 3);
+
+                int16x8_t res2;
+                filter4_ps_u16x8<coeff4>(in + 0, filter, offset, res0);
+                filter4_ps_u16x8<coeff4>(in + 1, filter, offset, res1);
+
+                store_s16x8xn<2>(d, dstStride, res);
+            }
+
+            src += 8;
+            dst += 8;
+        }
     }
-    int16x8_t vc3 = vld1q_s16(coeff);
-    const int32x4_t voffset = vdupq_n_s32(offset);
+}
+
+template<int coeffIdx, int width, int height>
+void inline interp8_vert_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                                 intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    const uint16x8_t filter =
+        vreinterpretq_u16_s16(vabsq_s16(vld1q_s16(X265_NS::g_lumaFiltercoeffIdx)));
+    uint32x4_t offset =
+        vdupq_n_u32((unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS);
 
-    int row, col;
-    for (row = 0; row < blkheight; row++)
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
     {
-        for (col = 0; col < width; col += 8)
+        const uint16_t *s = src;
+        int16_t *d = dst;
+
+        if (width == 12)
         {
-            int32x4_t vsum, vsum2;
+            uint16x8_t in11;
+            load_u16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
 
-            int16x8_t inputN;
-            for (int i = 0; i < N; i++)
+            for (int row = 0; row < height; row += 4)
             {
-                inputi = vreinterpretq_s16_u16(vld1q_u16(src + col + i));
+                load_u16x8xn<4>(s, srcStride, in + 7);
+
+                int16x8_t res4;
+                filter8_ps_u16x8<coeffIdx>(in + 0, res0, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 1, res1, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 2, res2, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 3, res3, offset, filter);
+
+                store_s16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
             }
 
-            vsum = voffset;
-            vsum2 = voffset;
+            s = src + 8;
+            d = dst + 8;
+        }
+
+        uint16x4_t in11;
+        load_u16x4xn<7>(s, srcStride, in);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_u16x4xn<4>(s, srcStride, in + 7);
 
-            vsum = vmlal_lane_s16(vsum, vget_low_s16(input0),
-                                  vget_low_s16(vc3), 0);
-            vsum2 = vmlal_high_lane_s16(vsum2, input0, vget_low_s16(vc3), 0);
+            int16x4_t res4;
+            filter8_ps_u16x4<coeffIdx>(in + 0, res0, offset, filter);
+            filter8_ps_u16x4<coeffIdx>(in + 1, res1, offset, filter);
+            filter8_ps_u16x4<coeffIdx>(in + 2, res2, offset, filter);
+            filter8_ps_u16x4<coeffIdx>(in + 3, res3, offset, filter);
 
-            vsum = vmlal_lane_s16(vsum, vget_low_s16(input1),
-                                  vget_low_s16(vc3), 1);
-            vsum2 = vmlal_high_lane_s16(vsum2, input1, vget_low_s16(vc3), 1);
+            store_s16x4xn<4>(d, dstStride, res);
+
+            in0 = in4;
+            in1 = in5;
+            in2 = in6;
+            in3 = in7;
+            in4 = in8;
+            in5 = in9;
+            in6 = in10;
 
-            vsum = vmlal_lane_s16(vsum, vget_low_s16(input2),
-                                  vget_low_s16(vc3), 2);
-            vsum2 = vmlal_high_lane_s16(vsum2, input2, vget_low_s16(vc3), 2);
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+    }
+    else if (width % 16 != 0)
+    {
+        for (int col = 0; col < width; col += 8)
+        {
+            const uint16_t *s = src;
+            int16_t *d = dst;
 
-            vsum = vmlal_lane_s16(vsum, vget_low_s16(input3),
-                                  vget_low_s16(vc3), 3);
-            vsum2 = vmlal_high_lane_s16(vsum2, input3, vget_low_s16(vc3), 3);
+            uint16x8_t in11;
+            load_u16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
 
-            if (N == 8)
+            for (int row = 0; row < height; row += 4)
             {
-                vsum = vmlal_lane_s16(vsum, vget_low_s16(input4), vget_high_s16(vc3), 0);
-                vsum2 = vmlal_high_lane_s16(vsum2, input4, vget_high_s16(vc3), 0);
+                load_u16x8xn<4>(s, srcStride, in + 7);
 
-                vsum = vmlal_lane_s16(vsum, vget_low_s16(input5), vget_high_s16(vc3), 1);
-                vsum2 = vmlal_high_lane_s16(vsum2, input5, vget_high_s16(vc3), 1);
+                int16x8_t res4;
+                filter8_ps_u16x8<coeffIdx>(in + 0, res0, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 1, res1, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 2, res2, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in + 3, res3, offset, filter);
 
-                vsum = vmlal_lane_s16(vsum, vget_low_s16(input6), vget_high_s16(vc3), 2);
-                vsum2 = vmlal_high_lane_s16(vsum2, input6, vget_high_s16(vc3), 2);
+                store_s16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
 
-                vsum = vmlal_lane_s16(vsum, vget_low_s16(input7), vget_high_s16(vc3), 3);
-                vsum2 = vmlal_high_lane_s16(vsum2, input7, vget_high_s16(vc3), 3);
+                s += 4 * srcStride;
+                d += 4 * dstStride;
             }
 
-            int16x4_t res_lo = vshrn_n_s32(vsum, SHIFT_INTERP_PS);
-            int16x4_t res_hi = vshrn_n_s32(vsum2, SHIFT_INTERP_PS);
-            vst1q_s16(dst + col, vcombine_s16(res_lo, res_hi));
+            src += 8;
+            dst += 8;
         }
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 16)
+        {
+            const uint16_t *s = src;
+            int16_t *d = dst;
 
-        src += srcStride;
-        dst += dstStride;
+            uint16x8_t in011, in111;
+            load_u16x8xn<7>(s + 0, srcStride, in0);
+            load_u16x8xn<7>(s + 8, srcStride, in1);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_u16x8xn<4>(s + 0, srcStride, in0 + 7);
+                load_u16x8xn<4>(s + 8, srcStride, in1 + 7);
+
+                int16x8_t res04, res14;
+                filter8_ps_u16x8<coeffIdx>(in0 + 0, res00, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in0 + 1, res01, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in0 + 2, res02, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in0 + 3, res03, offset, filter);
+
+                filter8_ps_u16x8<coeffIdx>(in1 + 0, res10, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in1 + 1, res11, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in1 + 2, res12, offset, filter);
+                filter8_ps_u16x8<coeffIdx>(in1 + 3, res13, offset, filter);
+
+                store_s16x8xn<4>(d + 0, dstStride, res0);
+                store_s16x8xn<4>(d + 8, dstStride, res1);
+
+                in00 = in04;
+                in01 = in05;
+                in02 = in06;
+                in03 = in07;
+                in04 = in08;
+                in05 = in09;
+                in06 = in010;
+
+                in10 = in14;
+                in11 = in15;
+                in12 = in16;
+                in13 = in17;
+                in14 = in18;
+                in15 = in19;
+                in16 = in110;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            src += 16;
+            dst += 16;
+        }
+    }
+}
+
+template<bool coeff4>
+void inline filter4_sp_s16x4(const int16x4_t *s, const int16x4_t f,
+                             const int32x4_t offset, const uint16x4_t maxVal,
+                             uint16x4_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        int16x4_t sum03 = vadd_s16(s0, s3);
+        int16x4_t sum12 = vadd_s16(s1, s2);
+
+        int32x4_t sum = vmlal_n_s16(offset, sum12, 9);
+        sum = vsubw_s16(sum, sum03);
+
+        // We divided filter values by 4 so -2 from right shift.
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH - 2);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        int32x4_t sum = vmlal_lane_s16(offset, s0, f, 0);
+        sum = vmlal_lane_s16(sum, s1, f, 1);
+        sum = vmlal_lane_s16(sum, s2, f, 2);
+        sum = vmlal_lane_s16(sum, s3, f, 3);
+
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<bool coeff4>
+void inline filter4_sp_s16x8(const int16x8_t *s, const int16x4_t f,
+                             const int32x4_t offset, const uint16x8_t maxVal,
+                             uint16x8_t &d)
+{
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        // Filter values are divisible by 4, factor that out in order to only
+        // need a multiplication by 9 and a subtraction (which is a
+        // multiplication by -1).
+        int16x8_t sum03 = vaddq_s16(s0, s3);
+        int16x8_t sum12 = vaddq_s16(s1, s2);
+
+        int32x4_t sum_lo = vmlal_n_s16(offset, vget_low_s16(sum12), 9);
+        int32x4_t sum_hi = vmlal_n_s16(offset, vget_high_s16(sum12), 9);
+        sum_lo = vsubw_s16(sum_lo, vget_low_s16(sum03));
+        sum_hi = vsubw_s16(sum_hi, vget_high_s16(sum03));
+
+        // We divided filter values by 4 so -2 from right shift.
+        uint16x4_t d0 = vqshrun_n_s32(sum_lo,
+                                      IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH - 2);
+        uint16x4_t d1 = vqshrun_n_s32(sum_hi,
+                                      IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH - 2);
+        d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+    }
+    else
+    {
+        int32x4_t sum_lo = vmlal_lane_s16(offset, vget_low_s16(s0), f, 0);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s1), f, 1);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s2), f, 2);
+        sum_lo = vmlal_lane_s16(sum_lo, vget_low_s16(s3), f, 3);
+
+        int32x4_t sum_hi = vmlal_lane_s16(offset, vget_high_s16(s0), f, 0);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s1), f, 1);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s2), f, 2);
+        sum_hi = vmlal_lane_s16(sum_hi, vget_high_s16(s3), f, 3);
+
+        uint16x4_t d0 = vqshrun_n_s32(sum_lo,
+                                      IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        uint16x4_t d1 = vqshrun_n_s32(sum_hi,
+                                      IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        d = vminq_u16(vcombine_u16(d0, d1), maxVal);
+    }
+}
+
+template<bool coeff4, int width, int height>
+void inline interp4_vert_sp_neon(const int16_t *src, intptr_t srcStride, pixel *dst,
+                                 intptr_t dstStride, const int16_t coeffIdx)
+{
+    const int N_TAPS = 4;
+    const int shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    int16x4_t filter = vld1_s16(X265_NS::g_chromaFiltercoeffIdx);
+    int32x4_t offset;
+
+    if (coeff4)
+    {
+        // The right shift by 2 is needed because we will divide the filter values by 4.
+        offset = vdupq_n_s32(((1 << (shift - 1)) +
+                              (IF_INTERNAL_OFFS << IF_FILTER_PREC)) >> 2);
+    }
+    else
+    {
+        offset = vdupq_n_s32((1 << (shift - 1)) +
+                             (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+    }
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        if (width == 12 || width == 6)
+        {
+            const int n_store = width == 12 ? 8 : 6;
+            const int16_t *s = src;
+            uint16_t *d = dst;
+
+            int16x8_t in7;
+            load_s16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 3);
+
+                uint16x8_t res4;
+                filter4_sp_s16x8<coeff4>(in + 0, filter, offset, maxVal, res0);
+                filter4_sp_s16x8<coeff4>(in + 1, filter, offset, maxVal, res1);
+                filter4_sp_s16x8<coeff4>(in + 2, filter, offset, maxVal, res2);
+                filter4_sp_s16x8<coeff4>(in + 3, filter, offset, maxVal, res3);
+
+                store_u16xnxm<n_store, 4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (width == 6)
+            {
+                return;
+            }
+
+            src += 8;
+            dst += 8;
+        }
+        const int n_store = width > 4 ? 4 : width;
+
+        int16x4_t in7;
+        load_s16x4xn<3>(src, srcStride, in);
+        src += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_s16x4xn<4>(src, srcStride, in + 3);
+
+            uint16x4_t res4;
+            filter4_sp_s16x4<coeff4>(in + 0, filter, offset,
+                                     vget_low_u16(maxVal), res0);
+            filter4_sp_s16x4<coeff4>(in + 1, filter, offset,
+                                     vget_low_u16(maxVal), res1);
+            filter4_sp_s16x4<coeff4>(in + 2, filter, offset,
+                                     vget_low_u16(maxVal), res2);
+            filter4_sp_s16x4<coeff4>(in + 3, filter, offset,
+                                     vget_low_u16(maxVal), res3);
+
+            store_u16xnxm<n_store, 4>(dst, dstStride, res);
+
+            in0 = in4;
+            in1 = in5;
+            in2 = in6;
+
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_s16x4xn<2>(src, srcStride, in + 3);
+
+            uint16x4_t res2;
+            filter4_sp_s16x4<coeff4>(in + 0, filter, offset,
+                                     vget_low_u16(maxVal), res0);
+            filter4_sp_s16x4<coeff4>(in + 1, filter, offset,
+                                     vget_low_u16(maxVal), res1);
+
+            store_u16xnxm<n_store, 2>(dst, dstStride, res);
+        }
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 8)
+        {
+            const int16_t *s = src;
+            uint16_t *d = dst;
+
+            int16x8_t in7;
+            load_s16x8xn<3>(s, srcStride, in);
+            s += 3 * srcStride;
+
+            for (int row = 0; row + 4 <= height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 3);
+
+                uint16x8_t res4;
+                filter4_sp_s16x8<coeff4>(in + 0, filter, offset, maxVal, res0);
+                filter4_sp_s16x8<coeff4>(in + 1, filter, offset, maxVal, res1);
+                filter4_sp_s16x8<coeff4>(in + 2, filter, offset, maxVal, res2);
+                filter4_sp_s16x8<coeff4>(in + 3, filter, offset, maxVal, res3);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            if (height & 2)
+            {
+                load_s16x8xn<2>(s, srcStride, in + 3);
+
+                uint16x8_t res2;
+                filter4_sp_s16x8<coeff4>(in + 0, filter, offset, maxVal, res0);
+                filter4_sp_s16x8<coeff4>(in + 1, filter, offset, maxVal, res1);
+
+                store_u16x8xn<2>(d, dstStride, res);
+            }
+
+            src += 8;
+            dst += 8;
+        }
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_sp_s16x4(const int16x4_t *s, uint16x4_t &d, int32x4_t offset,
+                             int16x8_t filter, uint16x4_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        int16x4_t sum06 = vsub_s16(s6, s0);
+
+        int32x4_t sum12345 = vmlal_laneq_s16(offset, s1, filter, 1);
+        sum12345 = vmlal_laneq_s16(sum12345, s2, filter, 2);
+        sum12345 = vmlal_laneq_s16(sum12345, s3, filter, 3);
+        sum12345 = vmlal_laneq_s16(sum12345, s4, filter, 4);
+        sum12345 = vmlal_laneq_s16(sum12345, s5, filter, 5);
+
+        int32x4_t sum = vaddw_s16(sum12345, sum06);
+
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        d = vmin_u16(d, maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        int16x4_t sum07 = vadd_s16(s0, s7);
+        int16x4_t sum16 = vadd_s16(s1, s6);
+        int16x4_t sum25 = vadd_s16(s2, s5);
+        int16x4_t sum34 = vadd_s16(s3, s4);
+
+        int32x4_t sum12356 =  vmlal_laneq_s16(offset, sum16, filter, 1);
+        sum12356 = vmlal_laneq_s16(sum12356, sum25, filter, 2);
+        sum12356 = vmlal_laneq_s16(sum12356, sum34, filter, 3);
+
+        int32x4_t sum = vsubw_s16(sum12356, sum07);
+
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        d = vmin_u16(d, maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        int16x4_t sum17 = vsub_s16(s1, s7);
+
+        int32x4_t sum23456 = vmlal_laneq_s16(offset, s2, filter, 2);
+        sum23456 = vmlal_laneq_s16(sum23456, s3, filter, 3);
+        sum23456 = vmlal_laneq_s16(sum23456, s4, filter, 4);
+        sum23456 = vmlal_laneq_s16(sum23456, s5, filter, 5);
+        sum23456 = vmlal_laneq_s16(sum23456, s6, filter, 6);
+
+        int32x4_t sum = vaddw_s16(sum23456, sum17);
+
+        d = vqshrun_n_s32(sum, IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        d = vmin_u16(d, maxVal);
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_sp_s16x8(const int16x8_t *s, uint16x8_t &d, int32x4_t offset,
+                             int16x8_t filter, uint16x8_t maxVal)
+{
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        int16x8_t sum06 =  vsubq_s16(s6, s0);
+
+        int32x4_t sum12345_lo = vmlal_laneq_s16(offset, vget_low_s16(s1), filter, 1);
+        sum12345_lo = vmlal_laneq_s16(sum12345_lo, vget_low_s16(s2), filter, 2);
+        sum12345_lo = vmlal_laneq_s16(sum12345_lo, vget_low_s16(s3), filter, 3);
+        sum12345_lo = vmlal_laneq_s16(sum12345_lo, vget_low_s16(s4), filter, 4);
+        sum12345_lo = vmlal_laneq_s16(sum12345_lo, vget_low_s16(s5), filter, 5);
+
+        int32x4_t sum12345_hi = vmlal_laneq_s16(offset, vget_high_s16(s1), filter, 1);
+        sum12345_hi = vmlal_laneq_s16(sum12345_hi, vget_high_s16(s2), filter, 2);
+        sum12345_hi = vmlal_laneq_s16(sum12345_hi, vget_high_s16(s3), filter, 3);
+        sum12345_hi = vmlal_laneq_s16(sum12345_hi, vget_high_s16(s4), filter, 4);
+        sum12345_hi = vmlal_laneq_s16(sum12345_hi, vget_high_s16(s5), filter, 5);
+
+        int32x4_t sum_lo = vaddw_s16(sum12345_lo, vget_low_s16(sum06));
+        int32x4_t sum_hi = vaddw_s16(sum12345_hi, vget_high_s16(sum06));
+
+        uint16x4_t d_lo = vqshrun_n_s32(sum_lo,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        uint16x4_t d_hi = vqshrun_n_s32(sum_hi,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        int16x8_t sum07 = vaddq_s16(s0, s7);
+        int16x8_t sum16 = vaddq_s16(s1, s6);
+        int16x8_t sum25 = vaddq_s16(s2, s5);
+        int16x8_t sum34 = vaddq_s16(s3, s4);
+
+        int32x4_t sum123456_lo = vmlal_laneq_s16(offset, vget_low_s16(sum16), filter, 1);
+        sum123456_lo = vmlal_laneq_s16(sum123456_lo, vget_low_s16(sum25), filter, 2);
+        sum123456_lo = vmlal_laneq_s16(sum123456_lo, vget_low_s16(sum34), filter, 3);
+
+        int32x4_t sum123456_hi = vmlal_laneq_s16(offset, vget_high_s16(sum16), filter, 1);
+        sum123456_hi = vmlal_laneq_s16(sum123456_hi, vget_high_s16(sum25), filter, 2);
+        sum123456_hi = vmlal_laneq_s16(sum123456_hi, vget_high_s16(sum34), filter, 3);
+
+        int32x4_t sum_lo = vsubw_s16(sum123456_lo, vget_low_s16(sum07));
+        int32x4_t sum_hi = vsubw_s16(sum123456_hi, vget_high_s16(sum07));
+
+        uint16x4_t d_lo = vqshrun_n_s32(sum_lo,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        uint16x4_t d_hi = vqshrun_n_s32(sum_hi,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        int16x8_t sum17 = vsubq_s16(s1, s7);
+
+        int32x4_t sum23456_lo = vmlal_laneq_s16(offset, vget_low_s16(s2), filter, 2);
+        sum23456_lo = vmlal_laneq_s16(sum23456_lo, vget_low_s16(s3), filter, 3);
+        sum23456_lo = vmlal_laneq_s16(sum23456_lo, vget_low_s16(s4), filter, 4);
+        sum23456_lo = vmlal_laneq_s16(sum23456_lo, vget_low_s16(s5), filter, 5);
+        sum23456_lo = vmlal_laneq_s16(sum23456_lo, vget_low_s16(s6), filter, 6);
+
+        int32x4_t sum23456_hi = vmlal_laneq_s16(offset, vget_high_s16(s2), filter, 2);
+        sum23456_hi = vmlal_laneq_s16(sum23456_hi, vget_high_s16(s3), filter, 3);
+        sum23456_hi = vmlal_laneq_s16(sum23456_hi, vget_high_s16(s4), filter, 4);
+        sum23456_hi = vmlal_laneq_s16(sum23456_hi, vget_high_s16(s5), filter, 5);
+        sum23456_hi = vmlal_laneq_s16(sum23456_hi, vget_high_s16(s6), filter, 6);
+
+        int32x4_t sum_lo = vaddw_s16(sum23456_lo, vget_low_s16(sum17));
+        int32x4_t sum_hi = vaddw_s16(sum23456_hi, vget_high_s16(sum17));
+
+        uint16x4_t d_lo = vqshrun_n_s32(sum_lo,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+        uint16x4_t d_hi = vqshrun_n_s32(sum_hi,
+                                        IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH);
+
+        d = vminq_u16(vcombine_u16(d_lo, d_hi), maxVal);
+    }
+}
+
+template<int coeffIdx, int width, int height>
+void inline interp8_vert_sp_neon(const int16_t *src, intptr_t srcStride, pixel *dst,
+                                 intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    int shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+    const int16x8_t filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdx);
+    const int32x4_t offset = vdupq_n_s32((1 << (shift - 1)) +
+                                         (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    if (width % 8 != 0)
+    {
+        const int16_t *s = src;
+        uint16_t *d = dst;
+
+        if (width == 12)
+        {
+            int16x8_t in11;
+            load_s16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 7);
+
+                uint16x8_t res4;
+                filter8_sp_s16x8<coeffIdx>(in + 0, res0, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 1, res1, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 2, res2, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 3, res3, offset, filter, maxVal);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            s = src + 8;
+            d = dst + 8;
+        }
+
+        int16x4_t in11;
+        load_s16x4xn<7>(s, srcStride, in);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_s16x4xn<4>(s, srcStride, in + 7);
+
+            uint16x4_t res4;
+            filter8_sp_s16x4<coeffIdx>(in + 0, res0, offset, filter,
+                                       vget_low_u16(maxVal));
+            filter8_sp_s16x4<coeffIdx>(in + 1, res1, offset, filter,
+                                       vget_low_u16(maxVal));
+            filter8_sp_s16x4<coeffIdx>(in + 2, res2, offset, filter,
+                                       vget_low_u16(maxVal));
+            filter8_sp_s16x4<coeffIdx>(in + 3, res3, offset, filter,
+                                       vget_low_u16(maxVal));
+
+            store_u16x4xn<4>(d, dstStride, res);
+
+            in0 = in4;
+            in1 = in5;
+            in2 = in6;
+            in3 = in7;
+            in4 = in8;
+            in5 = in9;
+            in6 = in10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+    }
+    else if (width % 16 != 0)
+    {
+        const int16_t *s2 = src;
+        uint16_t *d2 = dst;
+        for (int col = 0; col < width; col += 8)
+        {
+            const int16_t *s = s2;
+            uint16_t *d = d2;
+
+            int16x8_t in11;
+            load_s16x8xn<7>(s, srcStride, in);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_s16x8xn<4>(s, srcStride, in + 7);
+
+                uint16x8_t res4;
+                filter8_sp_s16x8<coeffIdx>(in + 0, res0, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 1, res1, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 2, res2, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in + 3, res3, offset, filter, maxVal);
+
+                store_u16x8xn<4>(d, dstStride, res);
+
+                in0 = in4;
+                in1 = in5;
+                in2 = in6;
+                in3 = in7;
+                in4 = in8;
+                in5 = in9;
+                in6 = in10;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            s2 += 8;
+            d2 += 8;
+        }
+    }
+    else
+    {
+        for (int col = 0; col < width; col += 16)
+        {
+            const int16_t *s = src;
+            uint16_t *d = dst;
+
+            int16x8_t in011, in111;
+            load_s16x8xn<7>(s + 0, srcStride, in0);
+            load_s16x8xn<7>(s + 8, srcStride, in1);
+            s += 7 * srcStride;
+
+            for (int row = 0; row < height; row += 4)
+            {
+                load_s16x8xn<4>(s + 0, srcStride, in0 + 7);
+                load_s16x8xn<4>(s + 8, srcStride, in1 + 7);
+
+                uint16x8_t res04, res14;
+                filter8_sp_s16x8<coeffIdx>(in0 + 0, res00, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in0 + 1, res01, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in0 + 2, res02, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in0 + 3, res03, offset, filter, maxVal);
+
+                filter8_sp_s16x8<coeffIdx>(in1 + 0, res10, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in1 + 1, res11, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in1 + 2, res12, offset, filter, maxVal);
+                filter8_sp_s16x8<coeffIdx>(in1 + 3, res13, offset, filter, maxVal);
+
+                store_u16x8xn<4>(d + 0, dstStride, res0);
+                store_u16x8xn<4>(d + 8, dstStride, res1);
+
+                in00 = in04;
+                in01 = in05;
+                in02 = in06;
+                in03 = in07;
+                in04 = in08;
+                in05 = in09;
+                in06 = in010;
+
+                in10 = in14;
+                in11 = in15;
+                in12 = in16;
+                in13 = in17;
+                in14 = in18;
+                in15 = in19;
+                in16 = in110;
+
+                s += 4 * srcStride;
+                d += 4 * dstStride;
+            }
+
+            src += 16;
+            dst += 16;
+        }
+    }
+}
+
+template<int coeffIdx, int coeffIdy, int width, int height>
+void interp8_hv_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    const uint16x8_t h_filter =
+        vreinterpretq_u16_s16(vabsq_s16(vld1q_s16(X265_NS::g_lumaFiltercoeffIdx)));
+    const int16x8_t v_filter = vld1q_s16(X265_NS::g_lumaFiltercoeffIdy);
+    const uint32x4_t h_offset =
+        vdupq_n_u32((unsigned)-IF_INTERNAL_OFFS << SHIFT_INTERP_PS);
+    int shift = IF_FILTER_PREC + IF_INTERNAL_PREC - X265_DEPTH;
+    const int32x4_t v_offset = vdupq_n_s32((1 << (shift - 1)) +
+                                           (IF_INTERNAL_OFFS << IF_FILTER_PREC));
+    const uint16x8_t maxVal = vdupq_n_u16((1 << X265_DEPTH) - 1);
+
+    src -= (N_TAPS / 2 - 1) * srcStride + (N_TAPS / 2 - 1);
+
+    for (int col = 0; col + 8 <= width; col += 8)
+    {
+        const pixel *s = src;
+        pixel *d = dst;
+
+        uint16x8_t h_sN_TAPS;
+        int16x8_t v_s16;
+
+        load_u16x8xn<8>(s + 0 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s0, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 1 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s1, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 2 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s2, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 3 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s3, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 4 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s4, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 5 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s5, h_offset, h_filter);
+
+        load_u16x8xn<8>(s + 6 * srcStride, 1, h_s);
+        filter8_ps_u16x8<coeffIdx>(h_s, v_s6, h_offset, h_filter);
+
+        s += 7 * srcStride;
+
+        int row = 0;
+        if (coeffIdy == 1)
+        {
+            for (; row + 8 <= height; row += 8)
+            {
+                uint16x8_t res8;
+
+                load_u16x8xn<8>(s + 0 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s7, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 0, res0, v_offset, v_filter, maxVal);
+
+                load_u16x8xn<8>(s + 1 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s8, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 1, res1, v_offset, v_filter, maxVal);
+                v_s0 = v_s8;
+
+                load_u16x8xn<8>(s + 2 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s9, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 2, res2, v_offset, v_filter, maxVal);
+                v_s1 = v_s9;
+
+                load_u16x8xn<8>(s + 3 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s10, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 3, res3, v_offset, v_filter, maxVal);
+                v_s2 = v_s10;
+
+                load_u16x8xn<8>(s + 4 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s11, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 4, res4, v_offset, v_filter, maxVal);
+                v_s3 = v_s11;
+
+                load_u16x8xn<8>(s + 5 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s12, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 5, res5, v_offset, v_filter, maxVal);
+                v_s4 = v_s12;
+
+                load_u16x8xn<8>(s + 6 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s13, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 6, res6, v_offset, v_filter, maxVal);
+                v_s5 = v_s13;
+
+                load_u16x8xn<8>(s + 7 * srcStride, 1, h_s);
+                filter8_ps_u16x8<coeffIdx>(h_s, v_s14, h_offset, h_filter);
+                filter8_sp_s16x8<coeffIdy>(v_s + 7, res7, v_offset, v_filter, maxVal);
+                v_s6 = v_s14;
+
+                store_u16xnxm<8, 8>(d, dstStride, res);
+
+                s += 8 * srcStride;
+                d += 8 * dstStride;
+            }
+        }
+
+        for (; row < height; row += 4)
+        {
+            uint16x8_t res4;
+
+            load_u16x8xn<8>(s + 0 * srcStride, 1, h_s);
+            filter8_ps_u16x8<coeffIdx>(h_s, v_s7, h_offset, h_filter);
+            filter8_sp_s16x8<coeffIdy>(v_s + 0, res0, v_offset, v_filter, maxVal);
+            v_s0 = v_s4;
+
+            load_u16x8xn<8>(s + 1 * srcStride, 1, h_s);
+            filter8_ps_u16x8<coeffIdx>(h_s, v_s8, h_offset, h_filter);
+            filter8_sp_s16x8<coeffIdy>(v_s + 1, res1, v_offset, v_filter, maxVal);
+            v_s1 = v_s5;
+
+            load_u16x8xn<8>(s + 2 * srcStride, 1, h_s);
+            filter8_ps_u16x8<coeffIdx>(h_s, v_s9, h_offset, h_filter);
+            filter8_sp_s16x8<coeffIdy>(v_s + 2, res2, v_offset, v_filter, maxVal);
+            v_s2 = v_s6;
+
+            load_u16x8xn<8>(s + 3 * srcStride, 1, h_s);
+            filter8_ps_u16x8<coeffIdx>(h_s, v_s10, h_offset, h_filter);
+            filter8_sp_s16x8<coeffIdy>(v_s + 3, res3, v_offset, v_filter, maxVal);
+            v_s3 = v_s7;
+
+            store_u16xnxm<8, 4>(d, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += 8;
+        dst += 8;
+    }
+
+    if (width % 8 != 0)
+    {
+        uint16x4_t h_s0N_TAPS, h_s1N_TAPS;
+        int16x4_t v_s16;
+
+        load_u16x4xn<8>(src + 0 * srcStride, 1, h_s0);
+        filter8_ps_u16x4<coeffIdx>(h_s0, v_s0, h_offset, h_filter);
+
+        load_u16x4xn<8>(src + 1 * srcStride, 1, h_s1);
+        load_u16x4xn<8>(src + 2 * srcStride, 1, h_s0);
+
+        filter8_ps_u16x4<coeffIdx>(h_s1, v_s1, h_offset, h_filter);
+        filter8_ps_u16x4<coeffIdx>(h_s0, v_s2, h_offset, h_filter);
+
+        load_u16x4xn<8>(src + 3 * srcStride, 1, h_s1);
+        load_u16x4xn<8>(src + 4 * srcStride, 1, h_s0);
+
+        filter8_ps_u16x4<coeffIdx>(h_s1, v_s3, h_offset, h_filter);
+        filter8_ps_u16x4<coeffIdx>(h_s0, v_s4, h_offset, h_filter);
+
+        load_u16x4xn<8>(src + 5 * srcStride, 1, h_s1);
+        load_u16x4xn<8>(src + 6 * srcStride, 1, h_s0);
+
+        filter8_ps_u16x4<coeffIdx>(h_s1, v_s5, h_offset, h_filter);
+        filter8_ps_u16x4<coeffIdx>(h_s0, v_s6, h_offset, h_filter);
+
+        src += 7 * srcStride;
+
+        int row = 0;
+        for (; row + 8 <= height; row += 8)
+        {
+            uint16x4_t res8;
+
+            load_u16x4xn<8>(src + 0 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 1 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s7, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 0, res0, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s8, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 1, res1, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s0 = v_s8;
+
+            load_u16x4xn<8>(src + 2 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 3 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s9, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 2, res2, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s1 = v_s9;
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s10, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 3, res3, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s2 = v_s10;
+
+            load_u16x4xn<8>(src + 4 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 5 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s11, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 4, res4, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s3 = v_s11;
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s12, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 5, res5, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s4 = v_s12;
+
+            load_u16x4xn<8>(src + 6 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 7 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s13, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 6, res6, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s5 = v_s13;
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s14, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 7, res7, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s6 = v_s14;
+
+            store_u16xnxm<4, 8>(dst, dstStride, res);
+
+            src += 8 * srcStride;
+            dst += 8 * dstStride;
+        }
+
+        for (; row < height; row += 4)
+        {
+            uint16x4_t res4;
+
+            load_u16x4xn<8>(src + 0 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 1 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s7, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 0, res0, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s0 = v_s4;
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s8, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 1, res1, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s1 = v_s5;
+
+            load_u16x4xn<8>(src + 2 * srcStride, 1, h_s1);
+            load_u16x4xn<8>(src + 3 * srcStride, 1, h_s0);
+
+            filter8_ps_u16x4<coeffIdx>(h_s1, v_s9, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 2, res2, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s2 = v_s6;
+
+            filter8_ps_u16x4<coeffIdx>(h_s0, v_s10, h_offset, h_filter);
+            filter8_sp_s16x4<coeffIdy>(v_s + 3, res3, v_offset, v_filter,
+                                       vget_low_u16(maxVal));
+            v_s3 = v_s7;
+
+            store_u16xnxm<4, 4>(dst, dstStride, res);
+
+            v_s4 = v_s8;
+            v_s5 = v_s9;
+            v_s6 = v_s10;
+
+            src += 4 * srcStride;
+            dst += 4 * dstStride;
+        }
+    }
+}
+
+#endif // !HIGH_BIT_DEPTH
+}
+
+namespace X265_NS
+{
+
+template<int N, int width, int height>
+void interp_horiz_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_horiz_pp_neon<1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_horiz_pp_neon<2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_horiz_pp_neon<3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_horiz_pp_neon<true, width, height>(src, srcStride,
+                                                              dst, dstStride,
+                                                              coeffIdx);
+        default:
+            return interp4_horiz_pp_neon<false, width, height>(src, srcStride,
+                                                               dst, dstStride,
+                                                               coeffIdx);
+        }
     }
 }
 
-#else // HIGH_BIT_DEPTH
 template<int N, int width, int height>
-void interp_horiz_ps_neon(const uint8_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx,
-                          int isRowExt)
+void interp_horiz_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride, int coeffIdx, int isRowExt)
 {
     if (N == 8)
     {
@@ -2343,8 +5100,6 @@
     }
 }
 
-#endif // HIGH_BIT_DEPTH
-
 template<int N, int width, int height>
 void interp_vert_ss_neon(const int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)
 {
@@ -2365,73 +5120,21 @@
     }
     else
     {
-        return interp4_vert_ss_neon<width, height>(src, srcStride, dst,
-                                                   dstStride, coeffIdx);
-    }
-}
-
-#if HIGH_BIT_DEPTH
-
-template<int N, int width, int height>
-void interp_vert_pp_neon(const uint16_t *src, intptr_t srcStride, uint16_t *dst, intptr_t dstStride, int coeffIdx)
-{
-
-    const int16_t *c = (N == 4) ? g_chromaFiltercoeffIdx : g_lumaFiltercoeffIdx;
-    int offset = 1 << (IF_FILTER_PREC - 1);
-    const uint16_t maxVal = (1 << X265_DEPTH) - 1;
-
-    src -= (N / 2 - 1) * srcStride;
-    int16x8_t vc = vld1q_s16(c);
-    int32x4_t low_vc = vmovl_s16(vget_low_s16(vc));
-    int32x4_t high_vc = vmovl_s16(vget_high_s16(vc));
-
-    const int32x4_t voffset = vdupq_n_s32(offset);
-
-    int row, col;
-    for (row = 0; row < height; row++)
-    {
-        for (col = 0; col < width; col += 4)
+        switch (coeffIdx)
         {
-            int32x4_t vsum;
-
-            int32x4_t inputN;
-
-            for (int i = 0; i < N; i++)
-            {
-                uint16x4_t in_tmp = vld1_u16(src + col + i * srcStride);
-                inputi = vreinterpretq_s32_u32(vmovl_u16(in_tmp));
-            }
-            vsum = voffset;
-
-            vsum = vmlaq_laneq_s32(vsum, (input0), low_vc, 0);
-            vsum = vmlaq_laneq_s32(vsum, (input1), low_vc, 1);
-            vsum = vmlaq_laneq_s32(vsum, (input2), low_vc, 2);
-            vsum = vmlaq_laneq_s32(vsum, (input3), low_vc, 3);
-
-            if (N == 8)
-            {
-                vsum = vmlaq_laneq_s32(vsum, (input4), high_vc, 0);
-                vsum = vmlaq_laneq_s32(vsum, (input5), high_vc, 1);
-                vsum = vmlaq_laneq_s32(vsum, (input6), high_vc, 2);
-                vsum = vmlaq_laneq_s32(vsum, (input7), high_vc, 3);
-            }
-
-            uint16x4_t res = vqshrun_n_s32(vsum, IF_FILTER_PREC);
-            res = vmin_u16(res, vdup_n_u16(maxVal));
-            vst1_u16(dst + col, res);
+        case 4:
+            return interp4_vert_ss_neon<true, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx);
+        default:
+            return interp4_vert_ss_neon<false, width, height>(src, srcStride, dst,
+                                                              dstStride, coeffIdx);
         }
-        src += srcStride;
-        dst += dstStride;
     }
 }
 
-
-
-
-#else
-
 template<int N, int width, int height>
-void interp_vert_pp_neon(const uint8_t *src, intptr_t srcStride, uint8_t *dst, intptr_t dstStride, int coeffIdx)
+void interp_vert_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                         intptr_t dstStride, int coeffIdx)
 {
     if (N == 8)
     {
@@ -2464,66 +5167,9 @@
     }
 }
 
-#endif
-
-
-#if HIGH_BIT_DEPTH
-
-template<int N, int width, int height>
-void interp_vert_ps_neon(const uint16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)
-{
-    const int16_t *c = (N == 4) ? g_chromaFiltercoeffIdx : g_lumaFiltercoeffIdx;
-    int offset = (unsigned) - IF_INTERNAL_OFFS << SHIFT_INTERP_PS;
-    src -= (N / 2 - 1) * srcStride;
-
-    int16x8_t vc = vld1q_s16(c);
-    int32x4_t low_vc = vmovl_s16(vget_low_s16(vc));
-    int32x4_t high_vc = vmovl_s16(vget_high_s16(vc));
-
-    const int32x4_t voffset = vdupq_n_s32(offset);
-
-    int row, col;
-    for (row = 0; row < height; row++)
-    {
-        for (col = 0; col < width; col += 4)
-        {
-            int32x4_t vsum;
-
-            int32x4_t inputN;
-
-            for (int i = 0; i < N; i++)
-            {
-                uint16x4_t in_tmp = vld1_u16(src + col + i * srcStride);
-                inputi = vreinterpretq_s32_u32(vmovl_u16(in_tmp));
-            }
-            vsum = voffset;
-
-            vsum = vmlaq_laneq_s32(vsum, input0, low_vc, 0);
-            vsum = vmlaq_laneq_s32(vsum, input1, low_vc, 1);
-            vsum = vmlaq_laneq_s32(vsum, input2, low_vc, 2);
-            vsum = vmlaq_laneq_s32(vsum, input3, low_vc, 3);
-
-            if (N == 8)
-            {
-                int32x4_t vsum1 = vmulq_laneq_s32(input4, high_vc, 0);
-                vsum1 = vmlaq_laneq_s32(vsum1, input5, high_vc, 1);
-                vsum1 = vmlaq_laneq_s32(vsum1, input6, high_vc, 2);
-                vsum1 = vmlaq_laneq_s32(vsum1, input7, high_vc, 3);
-                vsum = vaddq_s32(vsum, vsum1);
-            }
-
-            vst1_s16(dst + col, vshrn_n_s32(vsum, SHIFT_INTERP_PS));
-        }
-
-        src += srcStride;
-        dst += dstStride;
-    }
-}
-
-#else
-
 template<int N, int width, int height>
-void interp_vert_ps_neon(const uint8_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)
+void interp_vert_ps_neon(const pixel *src, intptr_t srcStride, int16_t *dst,
+                         intptr_t dstStride, int coeffIdx)
 {
     if (N == 8)
     {
@@ -2556,131 +5202,245 @@
     }
 }
 
-#endif
-
-
+template<int N, int width, int height>
+void interp_vert_sp_neon(const int16_t *src, intptr_t srcStride, pixel *dst,
+                         intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_vert_sp_neon<1, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 2:
+            return interp8_vert_sp_neon<2, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 3:
+            return interp8_vert_sp_neon<3, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_vert_sp_neon<true, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx);
+        default:
+            return interp4_vert_sp_neon<false, width, height>(src, srcStride, dst,
+                                                              dstStride, coeffIdx);
+        }
+    }
+}
 
 #if HIGH_BIT_DEPTH
 template<int N, int width, int height>
-void interp_vert_sp_neon(const int16_t *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int coeffIdx)
+void interp_hv_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                       intptr_t dstStride, int idxX, int idxY)
 {
-    int headRoom = IF_INTERNAL_PREC - X265_DEPTH;
-    int shift = IF_FILTER_PREC + headRoom;
-    int offset = (1 << (shift - 1)) + (IF_INTERNAL_OFFS << IF_FILTER_PREC);
-    uint16_t maxVal = (1 << X265_DEPTH) - 1;
-    const int16_t *coeff = (N == 8 ? g_lumaFiltercoeffIdx : g_chromaFiltercoeffIdx);
-
-    src -= (N / 2 - 1) * srcStride;
-
-    int16x8_t vc = vld1q_s16(coeff);
-    int16x4_t low_vc = vget_low_s16(vc);
-    int16x4_t high_vc = vget_high_s16(vc);
-
-    const int32x4_t voffset = vdupq_n_s32(offset);
-    const int32x4_t vhr = vdupq_n_s32(-shift);
-
-    int row, col;
-    for (row = 0; row < height; row++)
+// Use the merged hv paths with Clang only as performance with GCC is worse than the
+// existing approach of doing horizontal and vertical interpolation separately.
+#ifdef __clang__
+    switch (idxX)
     {
-        for (col = 0; col < width; col += 8)
+    case 1:
+    {
+        switch (idxY)
         {
-            int32x4_t vsum1, vsum2;
-
-            int16x8_t inputN;
-
-            for (int i = 0; i < N; i++)
-            {
-                inputi = vld1q_s16(src + col + i * srcStride);
-            }
-            vsum1 = voffset;
-            vsum2 = voffset;
-
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input0), low_vc, 0);
-            vsum2 = vmlal_high_lane_s16(vsum2, input0, low_vc, 0);
-
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input1), low_vc, 1);
-            vsum2 = vmlal_high_lane_s16(vsum2, input1, low_vc, 1);
-
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input2), low_vc, 2);
-            vsum2 = vmlal_high_lane_s16(vsum2, input2, low_vc, 2);
-
-            vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input3), low_vc, 3);
-            vsum2 = vmlal_high_lane_s16(vsum2, input3, low_vc, 3);
-
-            if (N == 8)
-            {
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input4), high_vc, 0);
-                vsum2 = vmlal_high_lane_s16(vsum2, input4, high_vc, 0);
+        case 1:
+            return interp8_hv_pp_neon<1, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_hv_pp_neon<1, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_hv_pp_neon<1, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
 
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input5), high_vc, 1);
-                vsum2 = vmlal_high_lane_s16(vsum2, input5, high_vc, 1);
+        break;
+    }
+    case 2:
+    {
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_neon<2, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_hv_pp_neon<2, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_hv_pp_neon<2, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
 
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input6), high_vc, 2);
-                vsum2 = vmlal_high_lane_s16(vsum2, input6, high_vc, 2);
+        break;
+    }
+    case 3:
+    {
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_neon<3, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_hv_pp_neon<3, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_hv_pp_neon<3, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
 
-                vsum1 = vmlal_lane_s16(vsum1, vget_low_s16(input7), high_vc, 3);
-                vsum2 = vmlal_high_lane_s16(vsum2, input7, high_vc, 3);
-            }
+        break;
+    }
+    }
 
-            vsum1 = vshlq_s32(vsum1, vhr);
-            vsum2 = vshlq_s32(vsum2, vhr);
+#else // __clang__
 
-            int16x8_t vsum = vuzp1q_s16(vreinterpretq_s16_s32(vsum1),
-                                        vreinterpretq_s16_s32(vsum2));
-            vsum = vminq_s16(vsum, vdupq_n_s16(maxVal));
-            vsum = vmaxq_s16(vsum, vdupq_n_s16(0));
-            vst1q_u16(dst + col, vreinterpretq_u16_s16(vsum));
-        }
+    ALIGN_VAR_32(int16_t, immedwidth * (height + N - 1));
 
-        src += srcStride;
-        dst += dstStride;
-    }
+    interp_horiz_ps_neon<N, width, height>(src, srcStride, immed, width, idxX, 1);
+    interp_vert_sp_neon<N, width, height>(immed + (N / 2 - 1) * width, width, dst,
+                                          dstStride, idxY);
+#endif // __clang__
 }
 
-#else // if HIGH_BIT_DEPTH
+#else // HIGH_BIT_DEPTH
 
 template<int N, int width, int height>
-void interp_vert_sp_neon(const int16_t *src, intptr_t srcStride, uint8_t *dst,
-                         intptr_t dstStride, int coeffIdx)
+void interp_hv_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst,
+                       intptr_t dstStride, int idxX, int idxY)
 {
-    if (N == 8)
+// Use the merged hv paths with Clang only as performance with GCC is worse than the
+// existing approach of doing horizontal and vertical interpolation separately.
+#ifdef __clang__
+    switch (idxX)
     {
-        switch (coeffIdx)
+    case 1:
+    {
+        switch (idxY)
         {
         case 1:
-            return interp8_vert_sp_neon<1, width, height>(src, srcStride, dst,
-                                                          dstStride);
+            return interp8_hv_pp_neon<1, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
         case 2:
-            return interp8_vert_sp_neon<2, width, height>(src, srcStride, dst,
-                                                          dstStride);
+            return interp8_hv_pp_neon<1, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
         case 3:
-            return interp8_vert_sp_neon<3, width, height>(src, srcStride, dst,
-                                                          dstStride);
+            return interp8_hv_pp_neon<1, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
         }
+
+        break;
     }
-    else
+    case 2:
     {
-        return interp4_vert_sp_neon<width, height>(src, srcStride, dst,
-                                                   dstStride, coeffIdx);
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_neon<2, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_hv_pp_neon<2, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_hv_pp_neon<2, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
+
+        break;
     }
-}
+    case 3:
+    {
+        switch (idxY)
+        {
+        case 1:
+            return interp8_hv_pp_neon<3, 1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_hv_pp_neon<3, 2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_hv_pp_neon<3, 3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
 
-#endif // if HIGH_BIT_DEPTH
+        break;
+    }
+    }
 
-template<int N, int width, int height>
-void interp_hv_pp_neon(const pixel *src, intptr_t srcStride, pixel *dst, intptr_t dstStride, int idxX, int idxY)
-{
+#else // __clang__
     ALIGN_VAR_32(int16_t, immedwidth * (height + N - 1));
 
     interp_horiz_ps_neon<N, width, height>(src, srcStride, immed, width, idxX, 1);
-    interp_vert_sp_neon<N, width, height>(immed + (N / 2 - 1) * width, width, dst, dstStride, idxY);
+    interp_vert_sp_neon<N, width, height>(immed + (N / 2 - 1) * width, width, dst,
+                                          dstStride, idxY);
+#endif // __clang__
 }
 
+#endif // HIGH_BIT_DEPTH
 
+template<int width, int height>
+void filterPixelToShort_neon(const pixel *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride)
+{
+    const int shift = IF_INTERNAL_PREC - X265_DEPTH;
+    const int16x8_t off = vdupq_n_s16(IF_INTERNAL_OFFS);
 
+    for (int row = 0; row < height; row++)
+    {
+        int col = 0;
+        for (; col + 8 <= width; col += 8)
+        {
+            uint16x8_t in;
+
+#if HIGH_BIT_DEPTH
+            in = vld1q_u16(src + col);
+#else
+            in = vmovl_u8(vld1_u8(src + col));
+#endif
+
+            int16x8_t tmp = vreinterpretq_s16_u16(vshlq_n_u16(in, shift));
+            tmp = vsubq_s16(tmp, off);
+            vst1q_s16(dst + col, tmp);
+        }
 
+        for (; col + 4 <= width; col += 4)
+        {
+            uint16x4_t in;
 
+#if HIGH_BIT_DEPTH
+            in = vld1_u16(src + col);
+#else
+            in = vget_low_u16(vmovl_u8(vld1_u8(src + col)));
+#endif
 
+            int16x4_t tmp = vreinterpret_s16_u16(vshl_n_u16(in, shift));
+            tmp = vsub_s16(tmp, vget_low_s16(off));
+            vst1_s16(dst + col, tmp);
+        }
+
+        for (; col < width; col += 2)
+        {
+            uint16x4_t in;
+
+#if HIGH_BIT_DEPTH
+            in = vld1_u16(src + col);
+#else
+            in = vget_low_u16(vmovl_u8(vld1_u8(src + col)));
+#endif
+
+            int16x4_t tmp = vreinterpret_s16_u16(vshl_n_u16(in, shift));
+            tmp = vsub_s16(tmp, vget_low_s16(off));
+            store_s16x2xn<1>(dst + col, dstStride, &tmp);
+        }
+
+        src += srcStride;
+        dst += dstStride;
+    }
+}
 
 #define CHROMA_420(W, H) \
     p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_hpp = interp_horiz_pp_neon<4, W, H>; \
@@ -2725,12 +5485,10 @@
 
 void setupFilterPrimitives_neon(EncoderPrimitives &p)
 {
-#if !HIGH_BIT_DEPTH
     LUMA(4, 4);
     LUMA(4, 8);
     LUMA(4, 16);
     LUMA(12, 16);
-#endif
     LUMA(8, 4);
     LUMA(8, 8);
     LUMA(8, 16);
@@ -2753,7 +5511,6 @@
     LUMA(64, 48);
     LUMA(64, 64);
 
-#if !HIGH_BIT_DEPTH
     CHROMA_420(2, 4);
     CHROMA_420(2, 8);
     CHROMA_420(4, 2);
@@ -2762,7 +5519,6 @@
     CHROMA_420(4, 16);
     CHROMA_420(6, 8);
     CHROMA_420(12, 16);
-#endif
     CHROMA_420(8, 2);
     CHROMA_420(8, 4);
     CHROMA_420(8, 6);
@@ -2780,7 +5536,6 @@
     CHROMA_420(32, 24);
     CHROMA_420(32, 32);
 
-#if !HIGH_BIT_DEPTH
     CHROMA_422(2, 8);
     CHROMA_422(2, 16);
     CHROMA_422(4, 4);
@@ -2789,7 +5544,6 @@
     CHROMA_422(4, 32);
     CHROMA_422(6, 16);
     CHROMA_422(12, 32);
-#endif
     CHROMA_422(8, 4);
     CHROMA_422(8, 8);
     CHROMA_422(8, 12);
@@ -2807,12 +5561,10 @@
     CHROMA_422(32, 48);
     CHROMA_422(32, 64);
 
-#if !HIGH_BIT_DEPTH
     CHROMA_444(4, 4);
     CHROMA_444(4, 8);
     CHROMA_444(4, 16);
     CHROMA_444(12, 16);
-#endif
     CHROMA_444(8, 4);
     CHROMA_444(8, 8);
     CHROMA_444(8, 16);

x265-4.1.tar/source/common/aarch64/filter-prim.h -> x265-4.2.tar/source/common/aarch64/filter-prim.h Changed

@@ -7,6 +7,116 @@
 #include "primitives.h"
 #include "x265.h"
 
+#include <arm_neon.h>
+
+static inline int16x8_t vtbl2q_s32_s16(int32x4_t a, int32x4_t b, uint8x16_t index)
+{
+    uint8x16x2_t ab;
+
+    ab.val0 = vreinterpretq_u8_s32(a);
+    ab.val1 = vreinterpretq_u8_s32(b);
+
+    return vreinterpretq_s16_u8(vqtbl2q_u8(ab, index));
+}
+
+template<int coeffIdx>
+void inline filter8_s16x4(const int16x4_t *s, const int16x8_t filter,
+                          const int32x4_t c, int32x4_t &d)
+{
+    if (coeffIdx == 1)
+    {
+        d = vsubl_s16(s6, s0);
+        d = vaddq_s32(d, c);
+        d = vmlal_laneq_s16(d, s1, filter, 1);
+        d = vmlal_laneq_s16(d, s2, filter, 2);
+        d = vmlal_laneq_s16(d, s3, filter, 3);
+        d = vmlal_laneq_s16(d, s4, filter, 4);
+        d = vmlal_laneq_s16(d, s5, filter, 5);
+    }
+    else if (coeffIdx == 2)
+    {
+        int16x4_t sum07 = vadd_s16(s0, s7);
+        int16x4_t sum16 = vadd_s16(s1, s6);
+        int16x4_t sum25 = vadd_s16(s2, s5);
+        int16x4_t sum34 = vadd_s16(s3, s4);
+
+        int32x4_t sum12356 =  vmlal_laneq_s16(c, sum16, filter, 1);
+        sum12356 = vmlal_laneq_s16(sum12356, sum25, filter, 2);
+        sum12356 = vmlal_laneq_s16(sum12356, sum34, filter, 3);
+
+        d = vsubw_s16(sum12356, sum07);
+    }
+    else
+    {
+        d = vsubl_s16(s1, s7);
+        d = vaddq_s32(d, c);
+        d = vmlal_laneq_s16(d, s2, filter, 2);
+        d = vmlal_laneq_s16(d, s3, filter, 3);
+        d = vmlal_laneq_s16(d, s4, filter, 4);
+        d = vmlal_laneq_s16(d, s5, filter, 5);
+        d = vmlal_laneq_s16(d, s6, filter, 6);
+    }
+}
+
+template<int coeffIdx>
+void inline filter8_s16x8(const int16x8_t *s, const int16x8_t filter,
+                          const int32x4_t c, int32x4_t &d0, int32x4_t &d1)
+{
+    if (coeffIdx == 1)
+    {
+        d0 = vsubl_s16(vget_low_s16(s6), vget_low_s16(s0));
+        d0 = vaddq_s32(d0, c);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s1), filter, 1);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s2), filter, 2);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s3), filter, 3);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s4), filter, 4);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s5), filter, 5);
+
+        d1 = vsubl_s16(vget_high_s16(s6), vget_high_s16(s0));
+        d1 = vaddq_s32(d1, c);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s1), filter, 1);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s2), filter, 2);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s3), filter, 3);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s4), filter, 4);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s5), filter, 5);
+    }
+    else if (coeffIdx == 2)
+    {
+        int16x8_t sum07 = vaddq_s16(s0, s7);
+        int16x8_t sum16 = vaddq_s16(s1, s6);
+        int16x8_t sum25 = vaddq_s16(s2, s5);
+        int16x8_t sum34 = vaddq_s16(s3, s4);
+
+        int32x4_t sum123456_lo = vmlal_laneq_s16(c, vget_low_s16(sum16), filter, 1);
+        sum123456_lo = vmlal_laneq_s16(sum123456_lo, vget_low_s16(sum25), filter, 2);
+        sum123456_lo = vmlal_laneq_s16(sum123456_lo, vget_low_s16(sum34), filter, 3);
+
+        int32x4_t sum123456_hi = vmlal_laneq_s16(c, vget_high_s16(sum16), filter, 1);
+        sum123456_hi = vmlal_laneq_s16(sum123456_hi, vget_high_s16(sum25), filter, 2);
+        sum123456_hi = vmlal_laneq_s16(sum123456_hi, vget_high_s16(sum34), filter, 3);
+
+        d0 = vsubw_s16(sum123456_lo, vget_low_s16(sum07));
+        d1 = vsubw_s16(sum123456_hi, vget_high_s16(sum07));
+    }
+    else
+    {
+        int16x8_t sum17 = vsubq_s16(s1, s7);
+        d0 = vaddw_s16(c, vget_low_s16(sum17));
+        d1 = vaddw_s16(c, vget_high_s16(sum17));
+
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s2), filter, 2);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s3), filter, 3);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s4), filter, 4);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s5), filter, 5);
+        d0 = vmlal_laneq_s16(d0, vget_low_s16(s6), filter, 6);
+
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s2), filter, 2);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s3), filter, 3);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s4), filter, 4);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s5), filter, 5);
+        d1 = vmlal_laneq_s16(d1, vget_high_s16(s6), filter, 6);
+    }
+}
 
 namespace X265_NS
 {

x265-4.1.tar/source/common/aarch64/fun-decls.h -> x265-4.2.tar/source/common/aarch64/fun-decls.h Changed

@@ -162,13 +162,6 @@
 FUNCDEF_PU_MULT_16(void, sad_x4, neon_dotprod, const pixel*, const pixel*, const pixel*, const pixel*, const pixel*, intptr_t, int32_t*);
 FUNCDEF_PU(sse_t, pixel_sse_pp, neon_dotprod, const pixel*, intptr_t, const pixel*, intptr_t);
 
-void PFX(pixel_planecopy_cp_neon(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift));
-
-uint64_t PFX(pixel_var_8x8_neon(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_16x16_neon(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_32x32_neon(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_64x64_neon(const pixel* pix, intptr_t stride));
-
 void PFX(getResidual4_neon(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride));
 void PFX(getResidual8_neon(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride));
 void PFX(getResidual16_neon(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride));
@@ -228,16 +221,10 @@
 
 int PFX(psyCost_4x4_neon)(const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride);
 int PFX(psyCost_8x8_neon)(const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride);
-void PFX(weight_pp_neon)(const pixel* src, pixel* dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset);
 void PFX(weight_sp_neon)(const int16_t* src, pixel* dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset);
 int PFX(scanPosLast_neon)(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign, uint16_t *coeffFlag, uint8_t *coeffNum, int numSig, const uint16_t* scanCG4x4, const int trSize);
 uint32_t PFX(costCoeffNxN_neon)(const uint16_t *scan, const coeff_t *coeff, intptr_t trSize, uint16_t *absCoeff, const uint8_t *tabSigCtx, uint32_t scanFlagMask, uint8_t *baseCtx, int offset, int scanPosSigOff, int subPosBase);
 
-uint64_t PFX(pixel_var_8x8_sve2(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_16x16_sve2(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_32x32_sve2(const pixel* pix, intptr_t stride));
-uint64_t PFX(pixel_var_64x64_sve2(const pixel* pix, intptr_t stride));
-
 void PFX(getResidual16_sve2(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride));
 void PFX(getResidual32_sve2(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride));
 
@@ -258,4 +245,5 @@
 
 int PFX(psyCost_8x8_sve2)(const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride);
 void PFX(weight_sp_sve2)(const int16_t* src, pixel* dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset);
-int PFX(scanPosLast_sve2)(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign, uint16_t *coeffFlag, uint8_t *coeffNum, int numSig, const uint16_t* scanCG4x4, const int trSize);
+
+int PFX(scanPosLast_sve2_bitperm)(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign, uint16_t *coeffFlag, uint8_t *coeffNum, int numSig, const uint16_t* scanCG4x4, const int trSize);

x265-4.1.tar/source/common/aarch64/intrapred-prim.cpp -> x265-4.2.tar/source/common/aarch64/intrapred-prim.cpp Changed

@@ -4,6 +4,7 @@
 
 #if HAVE_NEON
 #include "arm64-utils.h"
+#include "mem-neon.h"
 #include <arm_neon.h>
 
 using namespace X265_NS;
@@ -399,6 +400,136 @@
         }
 }
 
+#if !HIGH_BIT_DEPTH
+void intra_pred_planar4_neon(pixel *dst, intptr_t dstStride, const pixel *srcPix,
+                             int /*dirMode*/, int /*bFilter*/)
+{
+    const int log2Size = 2;
+    const int blkSize = 1 << log2Size;
+
+    uint8x16_t src = vld1q_u8(srcPix + 1);
+
+    uint8x8_t above =
+        vreinterpret_u8_u32(vdup_laneq_u32(vreinterpretq_u32_u8(src), 0));
+
+    uint8x8_t topRight = vdup_laneq_u8(src, blkSize);
+    uint8x8_t bottomLeft = vdup_laneq_u8(src, 3 * blkSize);
+
+    const uint8_t c216 =
+    {
+        {3, 2, 1, 0, 3, 2, 1, 0, 1, 2, 3, 4, 1, 2, 3, 4},
+        {3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3}
+    };
+
+    const uint8x16_t c0 = vld1q_u8(c0);
+    const uint8x16_t c1 = vld1q_u8(c1);
+
+    uint16x8_t t = vmull_u8(topRight, vget_high_u8(c0));
+    t = vmlal_u8(t, above, vget_low_u8(c1));
+    t = vmlal_u8(t, bottomLeft, vget_high_u8(c1));
+
+    uint8x8_t index02 = vcreate_u8(0x0A0A0A0A08080808);
+    uint8x8_t left02 = vqtbl1_u8(src, index02);
+    uint16x8_t t02 = vmlal_u8(t, left02, vget_low_u8(c0));
+    uint8x8_t d02 = vrshrn_n_u16(t02, log2Size + 1);
+
+    uint8x8_t index13 = vcreate_u8(0x0B0B0B0B09090909);
+    uint8x8_t left13 = vqtbl1_u8(src, index13);
+    uint16x8_t t13 = vmlal_u8(t, left13, vget_low_u8(c0));
+    uint16x8_t sub_bottomLeft_above = vsubl_u8(bottomLeft, above);
+    t13 = vaddq_u16(t13, sub_bottomLeft_above);
+    uint8x8_t d13 = vrshrn_n_u16(t13, log2Size + 1);
+
+    store_u8x4_strided_xN<2>(dst + 0 * dstStride, 2 * dstStride, &d02);
+    store_u8x4_strided_xN<2>(dst + 1 * dstStride, 2 * dstStride, &d13);
+}
+#endif
+
+#if !HIGH_BIT_DEPTH
+void intra_pred_planar32_neon(pixel *dst, intptr_t dstStride, const pixel *srcPix,
+                              int /*dirMode*/, int /*bFilter*/)
+{
+    const int log2Size = 5;
+    const int blkSize = 1 << log2Size;
+
+    const pixel *src0 = srcPix + 1;
+    const pixel *src1 = srcPix + 2 * blkSize + 1;
+
+    uint8x8_t above0 = vld1_u8(src0 + 0 * 8);
+    uint8x8_t above1 = vld1_u8(src0 + 1 * 8);
+    uint8x8_t above2 = vld1_u8(src0 + 2 * 8);
+    uint8x8_t above3 = vld1_u8(src0 + 3 * 8);
+
+    uint8x8_t topRight = vdup_n_u8(src0blkSize);
+    uint8x8_t bottomLeft = vdup_n_u8(src1blkSize);
+
+    const uint8_t c232 =
+    {
+        {31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16,
+         15, 14, 13, 12, 11, 10,  9,  8,  7,  6,  5,  4,  3,  2,  1,  0},
+        { 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
+         17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32}
+    };
+
+    // left constant
+    const uint8x8_t l0 = vld1_u8(c0 + 0 * 8);
+    const uint8x8_t l1 = vld1_u8(c0 + 1 * 8);
+    const uint8x8_t l2 = vld1_u8(c0 + 2 * 8);
+    const uint8x8_t l3 = vld1_u8(c0 + 3 * 8);
+
+    // topRight constant
+    const uint8x8_t tR0 = vld1_u8(c1 + 0 * 8);
+    const uint8x8_t tR1 = vld1_u8(c1 + 1 * 8);
+    const uint8x8_t tR2 = vld1_u8(c1 + 2 * 8);
+    const uint8x8_t tR3 = vld1_u8(c1 + 3 * 8);
+
+    const uint16x8_t offset = vdupq_n_u16(blkSize);
+    const uint16x8_t offset_bottomLeft = vaddw_u8(offset, bottomLeft);
+
+    const uint8x8_t c31 = vdup_n_u8(31);
+
+    uint16x8_t t0 = vmlal_u8(offset_bottomLeft, topRight, tR0);
+    t0 = vmlal_u8(t0, above0, c31);
+
+    uint16x8_t t1 = vmlal_u8(offset_bottomLeft, topRight, tR1);
+    t1 = vmlal_u8(t1, above1, c31);
+
+    uint16x8_t t2 = vmlal_u8(offset_bottomLeft, topRight, tR2);
+    t2 = vmlal_u8(t2, above2, c31);
+
+    uint16x8_t t3 = vmlal_u8(offset_bottomLeft, topRight, tR3);
+    t3 = vmlal_u8(t3, above3, c31);
+
+    uint16x8_t sub_bottomLeft_above0 = vsubl_u8(bottomLeft, above0);
+    uint16x8_t sub_bottomLeft_above1 = vsubl_u8(bottomLeft, above1);
+    uint16x8_t sub_bottomLeft_above2 = vsubl_u8(bottomLeft, above2);
+    uint16x8_t sub_bottomLeft_above3 = vsubl_u8(bottomLeft, above3);
+
+    for (int y = 0; y < 32; y++)
+    {
+        uint8x8_t left = vdup_n_u8(src1y);
+
+        uint16x8_t r0 = vmlal_u8(t0, left, l0);
+        uint16x8_t r1 = vmlal_u8(t1, left, l1);
+        uint16x8_t r2 = vmlal_u8(t2, left, l2);
+        uint16x8_t r3 = vmlal_u8(t3, left, l3);
+
+        uint8x8_t d4;
+        d0 = vshrn_n_u16(r0, log2Size + 1);
+        d1 = vshrn_n_u16(r1, log2Size + 1);
+        d2 = vshrn_n_u16(r2, log2Size + 1);
+        d3 = vshrn_n_u16(r3, log2Size + 1);
+
+        store_u8x8xn<4>(dst + y * dstStride, 8, d);
+
+        t0 = vaddq_u16(t0, sub_bottomLeft_above0);
+        t1 = vaddq_u16(t1, sub_bottomLeft_above1);
+        t2 = vaddq_u16(t2, sub_bottomLeft_above2);
+        t3 = vaddq_u16(t3, sub_bottomLeft_above3);
+    }
+}
+#endif
+
 static void dcPredFilter(const pixel* above, const pixel* left, pixel* dst, intptr_t dststride, int size)
 {
     // boundary pixels processing
@@ -576,8 +707,10 @@
     p.cuBLOCK_32x32.intra_pred_allangs = all_angs_pred_neon<5>;
 
 #if !HIGH_BIT_DEPTH
+    p.cuBLOCK_4x4.intra_predPLANAR_IDX = intra_pred_planar4_neon;
     p.cuBLOCK_8x8.intra_predPLANAR_IDX = PFX(intra_pred_planar8_neon);
     p.cuBLOCK_16x16.intra_predPLANAR_IDX = PFX(intra_pred_planar16_neon);
+    p.cuBLOCK_32x32.intra_predPLANAR_IDX = intra_pred_planar32_neon;
 #endif
 
     p.cuBLOCK_4x4.intra_predDC_IDX = intra_pred_dc_neon<4>;

x265-4.1.tar/source/common/aarch64/loopfilter-prim.cpp -> x265-4.2.tar/source/common/aarch64/loopfilter-prim.cpp Changed

@@ -1,9 +1,10 @@
 #include "common.h"
 #include "loopfilter-prim.h"
+#include "mem-neon.h"
 
 #define PIXEL_MIN 0
 
-
+using namespace X265_NS;
 
 #if !(HIGH_BIT_DEPTH) && defined(HAVE_NEON)
 #include<arm_neon.h>
@@ -255,9 +256,138 @@
     }
 }
 
+void pelFilterLumaStrong_V_neon(pixel *src, intptr_t srcStep, intptr_t offset,
+                                int32_t tcP, int32_t tcQ)
+{
+    X265_CHECK(offset == 1, "Offset value must be 1 for LumaStrong Vertical\n");
+
+    src -= offset * 4;
+
+    const int16x8_t tc_vec = vcombine_s16(vdup_n_s16(tcP), vdup_n_s16(tcQ));
+    const int16x8_t neg_tc_vec = vnegq_s16(tc_vec);
+
+    static const uint8_t filter38 =
+    {
+        { 0, 2, 1, 2, 2, 1, 1, 0 },
+        { 0, 3, 1, 2, 2, 1, 3, 0 },
+        { 0, 1, 1, 2, 2, 1, 2, 0 },
+    };
+
+    const uint8x8_t f0 = vld1_u8(filter0);
+    const uint8x8_t f1 = vld1_u8(filter1);
+    const uint8x8_t f2 = vld1_u8(filter2);
+
+    // -1 index means value is zero because TBL instructions
+    // zero elements that have out of range indices.
+    const uint8x8_t idx0 = { 255, 0,   1, 2, 3,   4, 5, 255 };
+    const uint8x8_t idx1 = { 255, 1,   2, 3, 4,   5, 6, 255 };
+    const uint8x8_t idx2 = { 255, 2,   3, 4, 5,   6, 7, 255 };
+    const uint8x8_t idx3 = { 255, 3,   4, 5, 6, 255, 3, 255 };
+    const uint8x8_t idx4 = { 255, 4, 255, 1, 2,   3, 4, 255 };
+
+    const int16x8_t neg_shift = { 0, -3, -2, -3, -3, -2, -3, 0 };
+
+    for (int i = 0; i < UNIT_SIZE; i++, src += srcStep)
+    {
+        uint8x8_t s = vld1_u8(src);
+        uint8x8_t s0 = vtbl1_u8(s, idx0);
+        uint8x8_t s1 = vtbl1_u8(s, idx1);
+        uint8x8_t s2 = vtbl1_u8(s, idx2);
+        uint8x8_t s3 = vtbl1_u8(s, idx3);
+        uint8x8_t s4 = vtbl1_u8(s, idx4);
+
+        uint16x8_t s34 = vaddl_u8(s3, s4);
+        uint16x8_t sum = vmlal_u8(s34, s0, f0);
+        sum = vmlal_u8(sum, s1, f1);
+        sum = vmlal_u8(sum, s2, f2);
+
+        sum = vrshlq_u16(sum, neg_shift);
+        sum = vsubw_u8(sum, s1);
+        sum = vreinterpretq_u16_s16(
+            vminq_s16(tc_vec, vmaxq_s16(neg_tc_vec, vreinterpretq_s16_u16(sum))));
+
+        uint8x8_t d = vmovn_u16(sum);
+        d = vadd_u8(d, s);
+        vst1_u8(src, d);
+    }
 }
 
+void pelFilterLumaStrong_H_neon(pixel *src, intptr_t srcStep, intptr_t offset,
+                                int32_t tcP, int32_t tcQ)
+{
+    X265_CHECK(UNIT_SIZE == 4 && srcStep == 1,
+               "UNIT_SIZE must be 4 and srcStep must be 1 for LumaStrong Horizontal\n");
+
+    (void)srcStep;
+
+    const int16x8_t tc_vec = vcombine_s16(vdup_n_s16(tcP), vdup_n_s16(tcQ));
+    const int16x8_t neg_tc_vec = vnegq_s16(tc_vec);
+
+    uint8x8_t m0 = vld1_u8(src - 4 * offset);
+    uint8x8_t m1 = vld1_u8(src - 3 * offset);
+    uint8x8_t m2 = vld1_u8(src - 2 * offset);
+    uint8x8_t m3 = vld1_u8(src - 1 * offset);
+    uint8x8_t m4 = vld1_u8(src - 0 * offset);
+    uint8x8_t m5 = vld1_u8(src + 1 * offset);
+    uint8x8_t m6 = vld1_u8(src + 2 * offset);
+    uint8x8_t m7 = vld1_u8(src + 3 * offset);
+
+    uint8x8_t m12 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m1), vreinterpret_u32_u8(m2)));
+    uint8x8_t m23 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m2), vreinterpret_u32_u8(m3)));
+    uint8x8_t m34 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m3), vreinterpret_u32_u8(m4)));
+    uint8x8_t m45 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m4), vreinterpret_u32_u8(m5)));
+    uint8x8_t m56 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m5), vreinterpret_u32_u8(m6)));
+
+    // src-1 * offset, src0 * offset
+    uint16x8_t p0 = vaddl_u8(m23, m34);
+    p0 = vaddw_u8(p0, m45);
+    uint16x8_t t0 = vshlq_n_u16(p0, 1);
+    uint16x8_t t1 = vaddl_u8(m12, m56);
+    uint16x8_t t01 = vaddq_u16(t0, t1);
+    t01 = vrshrq_n_u16(t01, 3);
+    t01 = vsubw_u8(t01, m34);
+    t01 = vreinterpretq_u16_s16(
+        vminq_s16(tc_vec, vmaxq_s16(neg_tc_vec, vreinterpretq_s16_u16(t01))));
+    uint8x8_t d01 = vmovn_u16(t01);
+    d01 = vadd_u8(d01, m34);
+    store_u8x4_strided_xN<2>(&src-1 * offset, 1 * offset, &d01);
+
+    uint8x8_t m16 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m1), vreinterpret_u32_u8(m6)));
+    uint8x8_t m25 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m2), vreinterpret_u32_u8(m5)));
+
+    // src-2 * offset, src1 * offset
+    uint16x8_t p1 = vaddw_u8(p0, m16);
+    uint16x8_t t23 = vrshrq_n_u16(p1, 2);
+    t23 = vsubw_u8(t23, m25);
+    t23 = vreinterpretq_u16_s16(
+        vminq_s16(tc_vec, vmaxq_s16(neg_tc_vec, vreinterpretq_s16_u16(t23))));
+    uint8x8_t d23 = vmovn_u16(t23);
+    d23 = vadd_u8(d23, m25);
+    store_u8x4_strided_xN<2>(&src-2 * offset, 3 * offset, &d23);
+
+    uint8x8_t m07 =
+        vreinterpret_u8_u32(vzip1_u32(vreinterpret_u32_u8(m0), vreinterpret_u32_u8(m7)));
+
+    // src-3 * offset, src2 * offset
+    uint16x8_t p2 = vaddl_u8(m07, m16);
+    uint16x8_t t45 = vmlaq_n_u16(p1, p2, 2);
+    t45 = vrshrq_n_u16(t45, 3);
+    t45 = vsubw_u8(t45, m16);
+    t45 = vreinterpretq_u16_s16(
+        vminq_s16(tc_vec, vmaxq_s16(neg_tc_vec, vreinterpretq_s16_u16(t45))));
+    uint8x8_t d45 = vmovn_u16(t45);
+    d45 = vadd_u8(d45, m16);
+    store_u8x4_strided_xN<2>(&src-3 * offset, 5 * offset, &d45);
+}
 
+} // namespace
 
 namespace X265_NS
 {
@@ -273,6 +403,8 @@
     p.saoCuOrgB0 = processSaoCUB0_neon;
     p.sign = calSign_neon;
 
+    p.pelFilterLumaStrong0 = pelFilterLumaStrong_V_neon;
+    p.pelFilterLumaStrong1 = pelFilterLumaStrong_H_neon;
 }

x265-4.1.tar/source/common/aarch64/mc-a-sve2.S -> x265-4.2.tar/source/common/aarch64/mc-a-sve2.S Changed

@@ -298,627 +298,3 @@
 pixel_avg_pp_64xN_sve2 32
 pixel_avg_pp_64xN_sve2 48
 pixel_avg_pp_64xN_sve2 64
-
-// void addAvg(const int16_t* src0, const int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)
-
-.macro addAvg_2xN_sve2 h
-function PFX(addAvg_2x\h\()_sve2)
-    ptrue           p0.s, vl2
-    ptrue           p1.h, vl4
-    ptrue           p2.h, vl2
-.rept \h / 2
-    ld1rw           {z0.s}, p0/z, x0
-    ld1rw           {z1.s}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    ld1rw           {z2.s}, p0/z, x0
-    ld1rw           {z3.s}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p1/m, z0.h, z1.h
-    add             z2.h, p1/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p2, x2
-    add             x2, x2, x5
-    st1b            {z2.h}, p2, x2
-    add             x2, x2, x5
-.endr
-    ret
-endfunc
-.endm
-
-addAvg_2xN_sve2 4
-addAvg_2xN_sve2 8
-addAvg_2xN_sve2 16
-
-.macro addAvg_6xN_sve2 h
-function PFX(addAvg_6x\h\()_sve2)
-    mov             w12, #\h / 2
-    ptrue           p0.b, vl16
-    ptrue           p2.h, vl6
-.Loop_sve2_addavg_6x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    ld1b            {z2.b}, p0/z, x0
-    ld1b            {z3.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    add             z2.h, p0/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    sqrshrnb        z2.b, z2.h, #7
-    add             z0.b, z0.b, #0x80
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p2, x2
-    add             x2, x2, x5
-    st1b            {z2.h}, p2, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_sve2_addavg_6x\h
-    ret
-endfunc
-.endm
-
-addAvg_6xN_sve2 8
-addAvg_6xN_sve2 16
-
-.macro addAvg_8xN_sve2 h
-function PFX(addAvg_8x\h\()_sve2)
-    ptrue           p0.b, vl16
-.rept \h / 2
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    ld1b            {z2.b}, p0/z, x0
-    ld1b            {z3.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    add             z2.h, p0/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    st1b            {z2.h}, p0, x2
-    add             x2, x2, x5
-.endr
-    ret
-endfunc
-.endm
-
-.macro addAvg_8xN1_sve2 h
-function PFX(addAvg_8x\h\()_sve2)
-    mov             w12, #\h / 2
-    ptrue           p0.b, vl16
-.Loop_sve2_addavg_8x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    ld1b            {z2.b}, p0/z, x0
-    ld1b            {z3.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    add             z2.h, p0/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    st1b            {z2.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_sve2_addavg_8x\h
-    ret
-endfunc
-.endm
-
-addAvg_8xN_sve2 2
-addAvg_8xN_sve2 4
-addAvg_8xN_sve2 6
-addAvg_8xN_sve2 8
-addAvg_8xN_sve2 12
-addAvg_8xN_sve2 16
-addAvg_8xN1_sve2 32
-addAvg_8xN1_sve2 64
-
-.macro addAvg_12xN_sve2 h
-function PFX(addAvg_12x\h\()_sve2)
-    mov             w12, #\h
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_12x\h
-    ptrue           p0.b, vl16
-    ptrue           p1.b, vl8
-.Loop_sve2_addavg_12x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    ld1b            {z2.b}, p1/z, x0, #1, mul vl
-    ld1b            {z3.b}, p1/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    add             z2.h, p1/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z2.h}, p1, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_sve2_addavg_12x\h
-    ret
-.vl_gt_16_addAvg_12x\h\():
-    mov             x10, #24
-    mov             x11, #0
-    whilelt         p0.b, x11, x10
-.Loop_sve2_gt_16_addavg_12x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_sve2_gt_16_addavg_12x\h
-    ret
-endfunc
-.endm
-
-addAvg_12xN_sve2 16
-addAvg_12xN_sve2 32
-
-.macro addAvg_16xN_sve2 h
-function PFX(addAvg_16x\h\()_sve2)
-    mov             w12, #\h
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_16x\h
-    ptrue           p0.b, vl16
-.Loop_eq_16_sve2_addavg_16x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    ld1b            {z2.b}, p0/z, x0, #1, mul vl
-    ld1b            {z3.b}, p0/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    add             z2.h, p0/m, z2.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z2.h}, p0, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_eq_16_sve2_addavg_16x\h
-    ret
-.vl_gt_16_addAvg_16x\h\():
-    cmp             x9, #32
-    bgt             .vl_gt_32_addAvg_16x\h
-    ptrue           p0.b, vl32
-.Loop_gt_16_sve2_addavg_16x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p1, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_16_sve2_addavg_16x\h
-    ret
-.vl_gt_32_addAvg_16x\h\():
-    mov             x10, #48
-    mov             x11, #0
-    whilelt         p0.b, x11, x10
-.Loop_gt_32_sve2_addavg_16x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_32_sve2_addavg_16x\h
-    ret
-endfunc
-.endm
-
-addAvg_16xN_sve2 4
-addAvg_16xN_sve2 8
-addAvg_16xN_sve2 12
-addAvg_16xN_sve2 16
-addAvg_16xN_sve2 24
-addAvg_16xN_sve2 32
-addAvg_16xN_sve2 64
-
-.macro addAvg_24xN_sve2 h
-function PFX(addAvg_24x\h\()_sve2)
-    mov             w12, #\h
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_24x\h
-    addAvg_start
-.Loop_eq_16_sve2_addavg_24x\h\():
-    sub             w12, w12, #1
-    ld1             {v0.16b-v2.16b}, x0, x3
-    ld1             {v3.16b-v5.16b}, x1, x4
-    addavg_1        v0, v3
-    addavg_1        v1, v4
-    addavg_1        v2, v5
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    sqxtun          v2.8b, v2.8h
-    st1             {v0.8b-v2.8b}, x2, x5
-    cbnz            w12, .Loop_eq_16_sve2_addavg_24x\h
-    ret
-.vl_gt_16_addAvg_24x\h\():
-    cmp             x9, #48
-    bgt             .vl_gt_48_addAvg_24x\h
-    ptrue           p0.b, vl32
-    ptrue           p1.b, vl16
-.Loop_gt_16_sve2_addavg_24x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p1/z, x0, #1, mul vl
-    ld1b            {z2.b}, p0/z, x1
-    ld1b            {z3.b}, p1/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z2.h
-    add             z1.h, p1/m, z1.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p1, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_16_sve2_addavg_24x\h
-    ret
-.vl_gt_48_addAvg_24x\h\():
-    mov             x10, #48
-    mov             x11, #0
-    whilelt         p0.b, x11, x10
-.Loop_gt_48_sve2_addavg_24x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z2.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z2.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_48_sve2_addavg_24x\h
-    ret
-endfunc
-.endm
-
-addAvg_24xN_sve2 32
-addAvg_24xN_sve2 64
-
-.macro addAvg_32xN_sve2 h
-function PFX(addAvg_32x\h\()_sve2)
-    mov             w12, #\h
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_32x\h
-    ptrue           p0.b, vl16
-.Loop_eq_16_sve2_addavg_32x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x0, #1, mul vl
-    ld1b            {z2.b}, p0/z, x0, #2, mul vl
-    ld1b            {z3.b}, p0/z, x0, #3, mul vl
-    ld1b            {z4.b}, p0/z, x1
-    ld1b            {z5.b}, p0/z, x1, #1, mul vl
-    ld1b            {z6.b}, p0/z, x1, #2, mul vl
-    ld1b            {z7.b}, p0/z, x1, #3, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    add             z1.h, p0/m, z1.h, z5.h
-    add             z2.h, p0/m, z2.h, z6.h
-    add             z3.h, p0/m, z3.h, z7.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    sqrshrnb        z3.b, z3.h, #7
-    add             z3.b, z3.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p0, x2, #1, mul vl
-    st1b            {z2.h}, p0, x2, #2, mul vl
-    st1b            {z3.h}, p0, x2, #3, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_eq_16_sve2_addavg_32x\h
-    ret
-.vl_gt_16_addAvg_32x\h\():
-    cmp             x9, #48
-    bgt             .vl_gt_48_addAvg_32x\h
-    ptrue           p0.b, vl32
-.Loop_gt_eq_32_sve2_addavg_32x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x0, #1, mul vl
-    ld1b            {z2.b}, p0/z, x1
-    ld1b            {z3.b}, p0/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z2.h
-    add             z1.h, p0/m, z1.h, z3.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z1.b, z1.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p0, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_eq_32_sve2_addavg_32x\h
-    ret
-.vl_gt_48_addAvg_32x\h\():
-    ptrue           p0.b, vl64
-.Loop_eq_64_sve2_addavg_32x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z1.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_eq_64_sve2_addavg_32x\h
-    ret
-endfunc
-.endm
-
-addAvg_32xN_sve2 8
-addAvg_32xN_sve2 16
-addAvg_32xN_sve2 24
-addAvg_32xN_sve2 32
-addAvg_32xN_sve2 48
-addAvg_32xN_sve2 64
-
-function PFX(addAvg_48x64_sve2)
-    mov             w12, #64
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_48x64
-    addAvg_start
-    sub             x3, x3, #64
-    sub             x4, x4, #64
-.Loop_eq_16_sve2_addavg_48x64:
-    sub             w12, w12, #1
-    ld1             {v0.8h-v3.8h}, x0, #64
-    ld1             {v4.8h-v7.8h}, x1, #64
-    ld1             {v20.8h-v21.8h}, x0, x3
-    ld1             {v22.8h-v23.8h}, x1, x4
-    addavg_1        v0, v4
-    addavg_1        v1, v5
-    addavg_1        v2, v6
-    addavg_1        v3, v7
-    addavg_1        v20, v22
-    addavg_1        v21, v23
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    sqxtun          v1.8b, v2.8h
-    sqxtun2         v1.16b, v3.8h
-    sqxtun          v2.8b, v20.8h
-    sqxtun2         v2.16b, v21.8h
-    st1             {v0.16b-v2.16b}, x2, x5
-    cbnz            w12, .Loop_eq_16_sve2_addavg_48x64
-    ret
-.vl_gt_16_addAvg_48x64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_addAvg_48x64
-    ptrue           p0.b, vl32
-.Loop_gt_eq_32_sve2_addavg_48x64:
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x0, #1, mul vl
-    ld1b            {z2.b}, p0/z, x0, #2, mul vl
-    ld1b            {z4.b}, p0/z, x1
-    ld1b            {z5.b}, p0/z, x1, #1, mul vl
-    ld1b            {z6.b}, p0/z, x1, #2, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    add             z1.h, p0/m, z1.h, z5.h
-    add             z2.h, p0/m, z2.h, z6.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p0, x2, #1, mul vl
-    st1b            {z2.h}, p0, x2, #2, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_eq_32_sve2_addavg_48x64
-    ret
-.vl_gt_48_addAvg_48x64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_addAvg_48x64
-    ptrue           p0.b, vl64
-    ptrue           p1.b, vl32
-.Loop_gt_48_sve2_addavg_48x64:
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p1/z, x0, #1, mul vl
-    ld1b            {z4.b}, p0/z, x1
-    ld1b            {z5.b}, p1/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    add             z1.h, p1/m, z1.h, z5.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p1, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_48_sve2_addavg_48x64
-    ret
-.vl_gt_112_addAvg_48x64:
-    mov             x10, #96
-    mov             x11, #0
-    whilelt         p0.b, x11, x10
-.Loop_gt_112_sve2_addavg_48x64:
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z4.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_112_sve2_addavg_48x64
-    ret
-endfunc
-
-.macro addAvg_64xN_sve2 h
-function PFX(addAvg_64x\h\()_sve2)
-    mov             w12, #\h
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_addAvg_64x\h
-    addAvg_start
-    sub             x3, x3, #64
-    sub             x4, x4, #64
-.Loop_eq_16_sve2_addavg_64x\h\():
-    sub             w12, w12, #1
-    ld1             {v0.8h-v3.8h}, x0, #64
-    ld1             {v4.8h-v7.8h}, x1, #64
-    ld1             {v20.8h-v23.8h}, x0, x3
-    ld1             {v24.8h-v27.8h}, x1, x4
-    addavg_1        v0, v4
-    addavg_1        v1, v5
-    addavg_1        v2, v6
-    addavg_1        v3, v7
-    addavg_1        v20, v24
-    addavg_1        v21, v25
-    addavg_1        v22, v26
-    addavg_1        v23, v27
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    sqxtun          v1.8b, v2.8h
-    sqxtun2         v1.16b, v3.8h
-    sqxtun          v2.8b, v20.8h
-    sqxtun2         v2.16b, v21.8h
-    sqxtun          v3.8b, v22.8h
-    sqxtun2         v3.16b, v23.8h
-    st1             {v0.16b-v3.16b}, x2, x5
-    cbnz            w12, .Loop_eq_16_sve2_addavg_64x\h
-    ret
-.vl_gt_16_addAvg_64x\h\():
-    cmp             x9, #48
-    bgt             .vl_gt_48_addAvg_64x\h
-    ptrue           p0.b, vl32
-.Loop_gt_eq_32_sve2_addavg_64x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x0, #1, mul vl
-    ld1b            {z2.b}, p0/z, x0, #2, mul vl
-    ld1b            {z3.b}, p0/z, x0, #3, mul vl
-    ld1b            {z4.b}, p0/z, x1
-    ld1b            {z5.b}, p0/z, x1, #1, mul vl
-    ld1b            {z6.b}, p0/z, x1, #2, mul vl
-    ld1b            {z7.b}, p0/z, x1, #3, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    add             z1.h, p0/m, z1.h, z5.h
-    add             z2.h, p0/m, z2.h, z6.h
-    add             z3.h, p0/m, z3.h, z7.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    sqrshrnb        z2.b, z2.h, #7
-    add             z2.b, z2.b, #0x80
-    sqrshrnb        z3.b, z3.h, #7
-    add             z3.b, z3.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p0, x2, #1, mul vl
-    st1b            {z2.h}, p0, x2, #2, mul vl
-    st1b            {z3.h}, p0, x2, #3, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_eq_32_sve2_addavg_64x\h
-    ret
-.vl_gt_48_addAvg_64x\h\():
-    cmp             x9, #112
-    bgt             .vl_gt_112_addAvg_64x\h
-    ptrue           p0.b, vl64
-.Loop_gt_eq_48_sve2_addavg_64x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z1.b}, p0/z, x0, #1, mul vl
-    ld1b            {z4.b}, p0/z, x1
-    ld1b            {z5.b}, p0/z, x1, #1, mul vl
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    add             z1.h, p0/m, z1.h, z5.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    sqrshrnb        z1.b, z1.h, #7
-    add             z1.b, z1.b, #0x80
-    st1b            {z0.h}, p0, x2
-    st1b            {z1.h}, p0, x2, #1, mul vl
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_eq_48_sve2_addavg_64x\h
-    ret
-.vl_gt_112_addAvg_64x\h\():
-    ptrue           p0.b, vl128
-.Loop_gt_eq_128_sve2_addavg_64x\h\():
-    sub             w12, w12, #1
-    ld1b            {z0.b}, p0/z, x0
-    ld1b            {z4.b}, p0/z, x1
-    add             x0, x0, x3, lsl #1
-    add             x1, x1, x4, lsl #1
-    add             z0.h, p0/m, z0.h, z4.h
-    sqrshrnb        z0.b, z0.h, #7
-    add             z0.b, z0.b, #0x80
-    st1b            {z0.h}, p0, x2
-    add             x2, x2, x5
-    cbnz            w12, .Loop_gt_eq_128_sve2_addavg_64x\h
-    ret
-endfunc
-.endm
-
-addAvg_64xN_sve2 16
-addAvg_64xN_sve2 32
-addAvg_64xN_sve2 48
-addAvg_64xN_sve2 64

x265-4.1.tar/source/common/aarch64/mc-a.S -> x265-4.2.tar/source/common/aarch64/mc-a.S Changed

@@ -38,10 +38,13 @@
 .macro pixel_avg_pp_4xN_neon h
 function PFX(pixel_avg_pp_4x\h\()_neon)
 .rept \h
-    ld1             {v0.s}0, x2, x3
-    ld1             {v1.s}0, x4, x5
+    ldr             s0, x2
+    ldr             s1, x4
+    add             x2, x2, x3
+    add             x4, x4, x5
     urhadd          v2.8b, v0.8b, v1.8b
-    st1             {v2.s}0, x0, x1
+    str             s2, x0
+    add             x0, x0, x1
 .endr
     ret
 endfunc
@@ -73,13 +76,13 @@
     sub             x3, x3, #4
     sub             x5, x5, #4
 .rept 16
-    ld1             {v0.s}0, x2, #4
+    ldr             s0, x2, #4
     ld1             {v1.8b}, x2, x3
-    ld1             {v2.s}0, x4, #4
+    ldr             s2, x4, #4
     ld1             {v3.8b}, x4, x5
     urhadd          v4.8b, v0.8b, v2.8b
     urhadd          v5.8b, v1.8b, v3.8b
-    st1             {v4.s}0, x0, #4
+    str             s4, x0, #4
     st1             {v5.8b}, x0, x1
 .endr
     ret
@@ -214,344 +217,3 @@
 pixel_avg_pp_64xN_neon 32
 pixel_avg_pp_64xN_neon 48
 pixel_avg_pp_64xN_neon 64
-
-// void addAvg(const int16_t* src0, const int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)
-.macro addAvg_2xN h
-function PFX(addAvg_2x\h\()_neon)
-    addAvg_start
-.rept \h / 2
-    ldr             w10, x0
-    ldr             w11, x1
-    add             x0, x0, x3
-    add             x1, x1, x4
-    ldr             w12, x0
-    ldr             w13, x1
-    add             x0, x0, x3
-    add             x1, x1, x4
-    dup             v0.2s, w10
-    dup             v1.2s, w11
-    dup             v2.2s, w12
-    dup             v3.2s, w13
-    add             v0.4h, v0.4h, v1.4h
-    add             v2.4h, v2.4h, v3.4h
-    saddl           v0.4s, v0.4h, v30.4h
-    saddl           v2.4s, v2.4h, v30.4h
-    shrn            v0.4h, v0.4s, #7
-    shrn2           v0.8h, v2.4s, #7
-    sqxtun          v0.8b, v0.8h
-    st1             {v0.h}0, x2, x5
-    st1             {v0.h}2, x2, x5
-.endr
-    ret
-endfunc
-.endm
-
-addAvg_2xN 4
-addAvg_2xN 8
-addAvg_2xN 16
-
-.macro addAvg_4xN h
-function PFX(addAvg_4x\h\()_neon)
-    addAvg_start
-.rept \h / 2
-    ld1             {v0.8b}, x0, x3
-    ld1             {v1.8b}, x1, x4
-    ld1             {v2.8b}, x0, x3
-    ld1             {v3.8b}, x1, x4
-    add             v0.4h, v0.4h, v1.4h
-    add             v2.4h, v2.4h, v3.4h
-    saddl           v0.4s, v0.4h, v30.4h
-    saddl           v2.4s, v2.4h, v30.4h
-    shrn            v0.4h, v0.4s, #7
-    shrn2           v0.8h, v2.4s, #7
-    sqxtun          v0.8b, v0.8h
-    st1             {v0.s}0, x2, x5
-    st1             {v0.s}1, x2, x5
-.endr
-    ret
-endfunc
-.endm
-
-addAvg_4xN 2
-addAvg_4xN 4
-addAvg_4xN 8
-addAvg_4xN 16
-addAvg_4xN 32
-
-.macro addAvg_6xN h
-function PFX(addAvg_6x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h / 2
-    sub             x5, x5, #4
-.Loop_addavg_6x\h:
-    sub             w12, w12, #1
-    ld1             {v0.16b}, x0, x3
-    ld1             {v1.16b}, x1, x4
-    ld1             {v2.16b}, x0, x3
-    ld1             {v3.16b}, x1, x4
-    add             v0.8h, v0.8h, v1.8h
-    add             v2.8h, v2.8h, v3.8h
-    saddl           v16.4s, v0.4h, v30.4h
-    saddl2          v17.4s, v0.8h, v30.8h
-    saddl           v18.4s, v2.4h, v30.4h
-    saddl2          v19.4s, v2.8h, v30.8h
-    shrn            v0.4h, v16.4s, #7
-    shrn2           v0.8h, v17.4s, #7
-    shrn            v1.4h, v18.4s, #7
-    shrn2           v1.8h, v19.4s, #7
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    str             s0, x2, #4
-    st1             {v0.h}2, x2, x5
-    str             s1, x2, #4
-    st1             {v1.h}2, x2, x5
-    cbnz            w12, .Loop_addavg_6x\h
-    ret
-endfunc
-.endm
-
-addAvg_6xN 8
-addAvg_6xN 16
-
-.macro addAvg_8xN h
-function PFX(addAvg_8x\h\()_neon)
-    addAvg_start
-.rept \h / 2
-    ld1             {v0.16b}, x0, x3
-    ld1             {v1.16b}, x1, x4
-    ld1             {v2.16b}, x0, x3
-    ld1             {v3.16b}, x1, x4
-    add             v0.8h, v0.8h, v1.8h
-    add             v2.8h, v2.8h, v3.8h
-    saddl           v16.4s, v0.4h, v30.4h
-    saddl2          v17.4s, v0.8h, v30.8h
-    saddl           v18.4s, v2.4h, v30.4h
-    saddl2          v19.4s, v2.8h, v30.8h
-    shrn            v0.4h, v16.4s, #7
-    shrn2           v0.8h, v17.4s, #7
-    shrn            v1.4h, v18.4s, #7
-    shrn2           v1.8h, v19.4s, #7
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    st1             {v0.8b}, x2, x5
-    st1             {v1.8b}, x2, x5
-.endr
-    ret
-endfunc
-.endm
-
-.macro addAvg_8xN1 h
-function PFX(addAvg_8x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h / 2
-.Loop_addavg_8x\h:
-    sub             w12, w12, #1
-    ld1             {v0.16b}, x0, x3
-    ld1             {v1.16b}, x1, x4
-    ld1             {v2.16b}, x0, x3
-    ld1             {v3.16b}, x1, x4
-    add             v0.8h, v0.8h, v1.8h
-    add             v2.8h, v2.8h, v3.8h
-    saddl           v16.4s, v0.4h, v30.4h
-    saddl2          v17.4s, v0.8h, v30.8h
-    saddl           v18.4s, v2.4h, v30.4h
-    saddl2          v19.4s, v2.8h, v30.8h
-    shrn            v0.4h, v16.4s, #7
-    shrn2           v0.8h, v17.4s, #7
-    shrn            v1.4h, v18.4s, #7
-    shrn2           v1.8h, v19.4s, #7
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    st1             {v0.8b}, x2, x5
-    st1             {v1.8b}, x2, x5
-    cbnz            w12, .Loop_addavg_8x\h
-    ret
-endfunc
-.endm
-
-addAvg_8xN 2
-addAvg_8xN 4
-addAvg_8xN 6
-addAvg_8xN 8
-addAvg_8xN 12
-addAvg_8xN 16
-addAvg_8xN1 32
-addAvg_8xN1 64
-
-.macro addAvg_12xN h
-function PFX(addAvg_12x\h\()_neon)
-    addAvg_start
-    sub             x3, x3, #16
-    sub             x4, x4, #16
-    sub             x5, x5, #8
-    mov             w12, #\h
-.Loop_addAvg_12X\h\():
-    sub             w12, w12, #1
-    ld1             {v0.16b}, x0, #16
-    ld1             {v1.16b}, x1, #16
-    ld1             {v2.8b}, x0, x3
-    ld1             {v3.8b}, x1, x4
-    add             v0.8h, v0.8h, v1.8h
-    add             v2.4h, v2.4h, v3.4h
-    saddl           v16.4s, v0.4h, v30.4h
-    saddl2          v17.4s, v0.8h, v30.8h
-    saddl           v18.4s, v2.4h, v30.4h
-    shrn            v0.4h, v16.4s, #7
-    shrn2           v0.8h, v17.4s, #7
-    shrn            v1.4h, v18.4s, #7
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    st1             {v0.8b}, x2, #8
-    st1             {v1.s}0, x2, x5
-    cbnz            w12, .Loop_addAvg_12X\h
-    ret
-endfunc
-.endm
-
-addAvg_12xN 16
-addAvg_12xN 32
-
-.macro addAvg_16xN h
-function PFX(addAvg_16x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h
-.Loop_addavg_16x\h:
-    sub             w12, w12, #1
-    ld1             {v0.8h-v1.8h}, x0, x3
-    ld1             {v2.8h-v3.8h}, x1, x4
-    addavg_1        v0, v2
-    addavg_1        v1, v3
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    st1             {v0.16b}, x2, x5
-    cbnz            w12, .Loop_addavg_16x\h
-    ret
-endfunc
-.endm
-
-addAvg_16xN 4
-addAvg_16xN 8
-addAvg_16xN 12
-addAvg_16xN 16
-addAvg_16xN 24
-addAvg_16xN 32
-addAvg_16xN 64
-
-.macro addAvg_24xN h
-function PFX(addAvg_24x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h
-.Loop_addavg_24x\h\():
-    sub             w12, w12, #1
-    ld1             {v0.16b-v2.16b}, x0, x3
-    ld1             {v3.16b-v5.16b}, x1, x4
-    addavg_1        v0, v3
-    addavg_1        v1, v4
-    addavg_1        v2, v5
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    sqxtun          v2.8b, v2.8h
-    st1             {v0.8b-v2.8b}, x2, x5
-    cbnz            w12, .Loop_addavg_24x\h
-    ret
-endfunc
-.endm
-
-addAvg_24xN 32
-addAvg_24xN 64
-
-.macro addAvg_32xN h
-function PFX(addAvg_32x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h
-.Loop_addavg_32x\h\():
-    sub             w12, w12, #1
-    ld1             {v0.8h-v3.8h}, x0, x3
-    ld1             {v4.8h-v7.8h}, x1, x4
-    addavg_1        v0, v4
-    addavg_1        v1, v5
-    addavg_1        v2, v6
-    addavg_1        v3, v7
-    sqxtun          v0.8b, v0.8h
-    sqxtun          v1.8b, v1.8h
-    sqxtun          v2.8b, v2.8h
-    sqxtun          v3.8b, v3.8h
-    st1             {v0.8b-v3.8b}, x2, x5
-    cbnz            w12, .Loop_addavg_32x\h
-    ret
-endfunc
-.endm
-
-addAvg_32xN 8
-addAvg_32xN 16
-addAvg_32xN 24
-addAvg_32xN 32
-addAvg_32xN 48
-addAvg_32xN 64
-
-function PFX(addAvg_48x64_neon)
-    addAvg_start
-    sub             x3, x3, #64
-    sub             x4, x4, #64
-    mov             w12, #64
-.Loop_addavg_48x64:
-    sub             w12, w12, #1
-    ld1             {v0.8h-v3.8h}, x0, #64
-    ld1             {v4.8h-v7.8h}, x1, #64
-    ld1             {v20.8h-v21.8h}, x0, x3
-    ld1             {v22.8h-v23.8h}, x1, x4
-    addavg_1        v0, v4
-    addavg_1        v1, v5
-    addavg_1        v2, v6
-    addavg_1        v3, v7
-    addavg_1        v20, v22
-    addavg_1        v21, v23
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    sqxtun          v1.8b, v2.8h
-    sqxtun2         v1.16b, v3.8h
-    sqxtun          v2.8b, v20.8h
-    sqxtun2         v2.16b, v21.8h
-    st1             {v0.16b-v2.16b}, x2, x5
-    cbnz            w12, .Loop_addavg_48x64
-    ret
-endfunc
-
-.macro addAvg_64xN h
-function PFX(addAvg_64x\h\()_neon)
-    addAvg_start
-    mov             w12, #\h
-    sub             x3, x3, #64
-    sub             x4, x4, #64
-.Loop_addavg_64x\h\():
-    sub             w12, w12, #1
-    ld1             {v0.8h-v3.8h}, x0, #64
-    ld1             {v4.8h-v7.8h}, x1, #64
-    ld1             {v20.8h-v23.8h}, x0, x3
-    ld1             {v24.8h-v27.8h}, x1, x4
-    addavg_1        v0, v4
-    addavg_1        v1, v5
-    addavg_1        v2, v6
-    addavg_1        v3, v7
-    addavg_1        v20, v24
-    addavg_1        v21, v25
-    addavg_1        v22, v26
-    addavg_1        v23, v27
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    sqxtun          v1.8b, v2.8h
-    sqxtun2         v1.16b, v3.8h
-    sqxtun          v2.8b, v20.8h
-    sqxtun2         v2.16b, v21.8h
-    sqxtun          v3.8b, v22.8h
-    sqxtun2         v3.16b, v23.8h
-    st1             {v0.16b-v3.16b}, x2, x5
-    cbnz            w12, .Loop_addavg_64x\h
-    ret
-endfunc
-.endm
-
-addAvg_64xN 16
-addAvg_64xN 32
-addAvg_64xN 48
-addAvg_64xN 64

x265-4.1.tar/source/common/aarch64/mem-neon.h -> x265-4.2.tar/source/common/aarch64/mem-neon.h Changed

@@ -1,7 +1,8 @@
 /*****************************************************************************
- * Copyright (C) 2024 MulticoreWare, Inc
+ * Copyright (C) 2024-2025 MulticoreWare, Inc
  *
  * Authors: Hari Limaye <hari.limaye@arm.com>
+ *          Gerda Zsejke More <gerdazsejke.more@arm.com>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -28,6 +29,22 @@
 #include <cassert>
 #include <stdint.h>
 
+using namespace X265_NS;
+
+template<int N>
+static void inline store_u8x2_strided_xN(uint8_t *d, intptr_t stride,
+                                         const uint8x8_t *s)
+{
+    X265_CHECK(N % 2 == 0, "N should be divisible by 2");
+    for (int i = 0; i < N / 2; ++i)
+    {
+        vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(si), 0);
+        d += stride;
+        vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(si), 2);
+        d += stride;
+    }
+}
+
 // Load 4 bytes into the low half of a uint8x8_t, zero the upper half.
 static uint8x8_t inline load_u8x4x1(const uint8_t *s)
 {
@@ -57,6 +74,26 @@
     vst1_lane_u32((uint32_t *)d, vreinterpret_u32_u8(s), 0);
 }
 
+// Store 2 bytes from the low half of a uint8x8_t.
+static void inline store_u8x2x1(uint8_t *d, const uint8x8_t s)
+{
+    vst1_lane_u16((uint16_t *)d, vreinterpret_u16_u8(s), 0);
+}
+
+// Load 2 int16_t into a int16x8_t.
+static inline int16x8_t load_s16x2x1(const int16_t *p)
+{
+    int32x4_t ret = vld1q_lane_s32((const int32_t *)p, vdupq_n_s32(0), 0);
+
+    return vreinterpretq_s16_s32(ret);
+}
+
+// Store 2 uint16_t from the low half of a uint16x8_t.
+static inline void store_u16x2x1(const uint16_t *d, const uint16x8_t s)
+{
+    vst1q_lane_u32((uint32_t *)d, vreinterpretq_u32_u16(s), 0);
+}
+
 // Store N blocks of 32-bits from (N / 2) D-Registers.
 template<int N>
 static void inline store_u8x4_strided_xN(uint8_t *d, intptr_t stride,
@@ -89,8 +126,7 @@
 {
     for (int i = 0; i < N; ++i)
     {
-        dsti = vld1q_u8(src);
-        src += stride;
+        dsti = vld1q_u8(src + i * stride);
     }
 }
 
@@ -152,6 +188,17 @@
     }
 }
 
+template<int N, int M>
+static void inline store_u8xnxm_strided(uint8_t *dst, intptr_t dst_stride,
+                                        const uint8x8_t *src)
+{
+    switch (N)
+    {
+    case 2: return store_u8x2_strided_xN<M>(dst, dst_stride, src);
+    case 4: return store_u8x4_strided_xN<M>(dst, dst_stride, src);
+    }
+}
+
 template<int N>
 static void inline store_u8x16xn(uint8_t *dst, intptr_t dst_stride,
                                  const uint8x16_t *src)
@@ -186,6 +233,94 @@
 }
 
 template<int N>
+static void inline load_u16x4xn(const uint16_t *src, const intptr_t stride,
+                                uint16x4_t *dst)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        dsti = vld1_u16(src);
+        src += stride;
+    }
+}
+
+template<int N>
+static void inline load_u16x8xn(const uint16_t *src, const intptr_t stride,
+                                uint16x8_t *dst)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        dsti = vld1q_u16(src + i * stride);
+    }
+}
+
+template<int N>
+static void inline store_u16x2xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x4_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1_lane_u32((uint32_t *)dst, vreinterpret_u32_u16(srci), 0);
+        dst += dst_stride;
+    }
+}
+
+template<int N>
+static void inline store_u16x2xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x8_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1q_lane_u32((uint32_t *)dst, vreinterpretq_u32_u16(srci), 0);
+        dst += dst_stride;
+    }
+}
+
+template<int N>
+static void inline store_u16x4xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x4_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1_u16(dst, srci);
+        dst += dst_stride;
+    }
+}
+
+template<int N>
+static void inline store_u16x4xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x8_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1_u16(dst, vget_low_u16(srci));
+        dst += dst_stride;
+    }
+}
+
+template<int N>
+static void inline store_u16x6xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x8_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1_u16(dst, vget_low_u16(srci));
+        vst1q_lane_u32((uint32_t *)(dst + 4), vreinterpretq_u32_u16(srci), 2);
+        dst += dst_stride;
+    }
+}
+
+template<int N>
+static void inline store_u16x8xn(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x8_t *src)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        vst1q_u16(dst, srci);
+        dst += dst_stride;
+    }
+}
+
+template<int N>
 static void inline store_s16x2xn(int16_t *dst, intptr_t dst_stride,
                                  const int16x4_t *src)
 {
@@ -265,4 +400,39 @@
     }
 }
 
+template<int N, int M>
+static void inline store_s16xnxm(const int16x4_t *src, int16_t *dst,
+                                 intptr_t dst_stride)
+{
+    switch (N)
+    {
+    case 2: return store_s16x2xn<M>(dst, dst_stride, src);
+    case 4: return store_s16x4xn<M>(dst, dst_stride, src);
+    }
+}
+
+template<int N, int M>
+static void inline store_u16xnxm(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x8_t *src)
+{
+    switch (N)
+    {
+    case 2: return store_u16x2xn<M>(dst, dst_stride, src);
+    case 4: return store_u16x4xn<M>(dst, dst_stride, src);
+    case 6: return store_u16x6xn<M>(dst, dst_stride, src);
+    case 8: return store_u16x8xn<M>(dst, dst_stride, src);
+    }
+}
+
+template<int N, int M>
+static void inline store_u16xnxm(uint16_t *dst, intptr_t dst_stride,
+                                 const uint16x4_t *src)
+{
+    switch (N)
+    {
+    case 2: return store_u16x2xn<M>(dst, dst_stride, src);
+    case 4: return store_u16x4xn<M>(dst, dst_stride, src);
+    }
+}
+
 #endif // X265_COMMON_AARCH64_MEM_NEON_H

x265-4.1.tar/source/common/aarch64/neon-sve-bridge.h -> x265-4.2.tar/source/common/aarch64/neon-sve-bridge.h Changed

@@ -3,6 +3,7 @@
  *
  * Authors: Hari Limaye <hari.limaye@arm.com>
  *          Jonathan Wright <jonathan.wright@arm.com>
+ *          Gerda Zsejke More <gerdazsejke.more@arm.com>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -52,6 +53,24 @@
                                      svset_neonq_s16(svundef_s16(), y)));
 }
 
+#define x265_sdotq_lane_s16(sum, s0, f, lane)                               \
+        svget_neonq_s64(svdot_lane_s64(svset_neonq_s64(svundef_s64(), sum), \
+                                       svset_neonq_s16(svundef_s16(), s0),  \
+                                       svset_neonq_s16(svundef_s16(), f), lane))
+
+static inline uint64x2_t x265_udotq_u16(uint64x2_t acc, uint16x8_t x, uint16x8_t y)
+{
+    return svget_neonq_u64(svdot_u64(svset_neonq_u64(svundef_u64(), acc),
+                                     svset_neonq_u16(svundef_u16(), x),
+                                     svset_neonq_u16(svundef_u16(), y)));
+}
+
+static inline uint16x8_t x265_tblq_u16(uint16x8_t x, uint16x8_t idx)
+{
+    return svget_neonq_u16(svtbl_u16(svset_neonq_u16(svundef_u16(), x),
+                                     svset_neonq_u16(svundef_u16(), idx)));
+}
+
 static inline int8x16_t x265_sve_mask(const int x, const int endX,
                                       const int8x16_t in)
 {

x265-4.1.tar/source/common/aarch64/p2s-sve.S -> x265-4.2.tar/source/common/aarch64/p2s-sve.S Changed

x265-4.2.tar/source/common/aarch64/pixel-prim-neon-dotprod.cpp Added

@@ -0,0 +1,115 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Li Zhang <li.zhang2@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "pixel-prim.h"
+#include "mem-neon.h"
+
+#include <arm_neon.h>
+
+namespace
+{
+#if !HIGH_BIT_DEPTH
+template<int size>
+uint64_t pixel_var_neon_dotprod(const uint8_t *pix, intptr_t i_stride)
+{
+    if (size >= 16)
+    {
+        uint32x4_t sum2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+        uint32x4_t sqr2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+        for (int h = 0; h < size; h += 2)
+        {
+            for (int w = 0; w + 16 <= size; w += 16)
+            {
+                uint8x16_t s2;
+                load_u8x16xn<2>(pix + w, i_stride, s);
+
+                sum0 = vdotq_u32(sum0, s0, vdupq_n_u8(1));
+                sum1 = vdotq_u32(sum1, s1, vdupq_n_u8(1));
+
+                sqr0 = vdotq_u32(sqr0, s0, s0);
+                sqr1 = vdotq_u32(sqr1, s1, s1);
+            }
+
+            pix += 2 * i_stride;
+        }
+
+        sum0 = vaddq_u32(sum0, sum1);
+        sqr0 = vaddq_u32(sqr0, sqr1);
+
+        return vaddvq_u32(sum0) + (vaddlvq_u32(sqr0) << 32);
+    }
+    if (size == 8)
+    {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint32x2_t sqr = vdup_n_u32(0);
+
+        for (int h = 0; h < size; ++h)
+        {
+            uint8x8_t s = vld1_u8(pix);
+
+            sum = vaddw_u8(sum, s);
+            sqr = vdot_u32(sqr, s, s);
+
+            pix += i_stride;
+        }
+
+        return vaddvq_u16(sum) + (vaddlv_u32(sqr) << 32);
+    }
+    if (size == 4) {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint32x2_t sqr = vdup_n_u32(0);
+
+        for (int h = 0; h < size; h += 2)
+        {
+            uint8x8_t s = load_u8x4x2(pix, i_stride);
+
+            sum = vaddw_u8(sum, s);
+            sqr = vdot_u32(sqr, s, s);
+
+            pix += 2 * i_stride;
+        }
+
+        return vaddvq_u16(sum) + (vaddlv_u32(sqr) << 32);
+    }
+}
+#endif // !HIGH_BIT_DEPTH
+}
+
+namespace X265_NS
+{
+#if HIGH_BIT_DEPTH
+void setupPixelPrimitives_neon_dotprod(EncoderPrimitives &)
+{
+}
+#else // !HIGH_BIT_DEPTH
+void setupPixelPrimitives_neon_dotprod(EncoderPrimitives &p)
+{
+    p.cuBLOCK_4x4.var   = pixel_var_neon_dotprod<4>;
+    p.cuBLOCK_8x8.var   = pixel_var_neon_dotprod<8>;
+    p.cuBLOCK_16x16.var = pixel_var_neon_dotprod<16>;
+    p.cuBLOCK_32x32.var = pixel_var_neon_dotprod<32>;
+    p.cuBLOCK_64x64.var = pixel_var_neon_dotprod<64>;
+}
+#endif // HIGH_BIT_DEPTH
+}

x265-4.2.tar/source/common/aarch64/pixel-prim-sve.cpp Added

@@ -0,0 +1,141 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Li Zhang <li.zhang2@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "pixel-prim.h"
+#include "mem-neon.h"
+#include "neon-sve-bridge.h"
+
+#include <arm_neon.h>
+
+namespace
+{
+#if HIGH_BIT_DEPTH
+template<int size>
+uint64_t pixel_var_sve(const uint16_t *pix, intptr_t i_stride)
+{
+    if (size > 16)
+    {
+        uint64x2_t sum2 = { vdupq_n_u64(0), vdupq_n_u64(0) };
+        uint64x2_t sqr2 = { vdupq_n_u64(0), vdupq_n_u64(0) };
+
+        for (int h = 0; h < size; ++h)
+        {
+            for (int w = 0; w + 16 <= size; w += 16)
+            {
+                uint16x8_t s2;
+                load_u16x8xn<2>(pix + w, 8, s);
+
+                sum0 = x265_udotq_u16(sum0, s0, vdupq_n_u16(1));
+                sum1 = x265_udotq_u16(sum1, s1, vdupq_n_u16(1));
+
+                sqr0 = x265_udotq_u16(sqr0, s0, s0);
+                sqr1 = x265_udotq_u16(sqr1, s1, s1);
+            }
+
+            pix += i_stride;
+        }
+
+        sum0 = vaddq_u64(sum0, sum1);
+        sqr0 = vaddq_u64(sqr0, sqr1);
+
+        return vaddvq_u64(sum0) + (vaddvq_u64(sqr0) << 32);
+    }
+    if (size == 16)
+    {
+        uint16x8_t sum2 = { vdupq_n_u16(0), vdupq_n_u16(0) };
+        uint64x2_t sqr2 = { vdupq_n_u64(0), vdupq_n_u64(0) };
+
+        for (int h = 0; h < size; ++h)
+        {
+            uint16x8_t s2;
+            load_u16x8xn<2>(pix, 8, s);
+
+            sum0 = vaddq_u16(sum0, s0);
+            sum1 = vaddq_u16(sum1, s1);
+
+            sqr0 = x265_udotq_u16(sqr0, s0, s0);
+            sqr1 = x265_udotq_u16(sqr1, s1, s1);
+
+            pix += i_stride;
+        }
+
+        uint32x4_t sum_u32 = vpaddlq_u16(sum0);
+        sum_u32 = vpadalq_u16(sum_u32, sum1);
+        sqr0 = vaddq_u64(sqr0, sqr1);
+
+        return vaddvq_u32(sum_u32) + (vaddvq_u64(sqr0) << 32);
+    }
+    if (size == 8)
+    {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint64x2_t sqr = vdupq_n_u64(0);
+
+        for (int h = 0; h < size; ++h)
+        {
+            uint16x8_t s = vld1q_u16(pix);
+
+            sum = vaddq_u16(sum, s);
+            sqr = x265_udotq_u16(sqr, s, s);
+
+            pix += i_stride;
+        }
+
+        return vaddlvq_u16(sum) + (vaddvq_u64(sqr) << 32);
+    }
+    if (size == 4) {
+        uint16x4_t sum = vdup_n_u16(0);
+        uint32x4_t sqr = vdupq_n_u32(0);
+
+        for (int h = 0; h < size; ++h)
+        {
+            uint16x4_t s = vld1_u16(pix);
+
+            sum = vadd_u16(sum, s);
+            sqr = vmlal_u16(sqr, s, s);
+
+            pix += i_stride;
+        }
+
+        return vaddv_u16(sum) + (vaddlvq_u32(sqr) << 32);
+    }
+}
+#endif // HIGH_BIT_DEPTH
+}
+
+namespace X265_NS
+{
+#if HIGH_BIT_DEPTH
+void setupPixelPrimitives_sve(EncoderPrimitives &p)
+{
+    p.cuBLOCK_4x4.var   = pixel_var_sve<4>;
+    p.cuBLOCK_8x8.var   = pixel_var_sve<8>;
+    p.cuBLOCK_16x16.var = pixel_var_sve<16>;
+    p.cuBLOCK_32x32.var = pixel_var_sve<32>;
+    p.cuBLOCK_64x64.var = pixel_var_sve<64>;
+}
+#else // !HIGH_BIT_DEPTH
+void setupPixelPrimitives_sve(EncoderPrimitives &)
+{
+}
+#endif // HIGH_BIT_DEPTH
+}

x265-4.1.tar/source/common/aarch64/pixel-prim.cpp -> x265-4.2.tar/source/common/aarch64/pixel-prim.cpp Changed

@@ -19,818 +19,806 @@
 {
 
 
-/* SATD SA8D variants - based on x264 */
-static inline void SUMSUB_AB(int16x8_t &sum, int16x8_t &sub, const int16x8_t a, const int16x8_t b)
+static inline void sumsubq_s16(int16x8_t *sum, int16x8_t *sub, const int16x8_t a, const int16x8_t b)
 {
-    sum = vaddq_s16(a, b);
-    sub = vsubq_s16(a, b);
+    *sum = vaddq_s16(a, b);
+    *sub = vsubq_s16(a, b);
 }
 
-static inline void transpose_8h_8h(int16x8_t &t1, int16x8_t &t2,
-                                   const int16x8_t s1, const int16x8_t s2)
+static inline void transpose_s16_s16x2(int16x8_t *t1, int16x8_t *t2,
+                                       const int16x8_t s1, const int16x8_t s2)
 {
-    t1 = vtrn1q_s16(s1, s2);
-    t2 = vtrn2q_s16(s1, s2);
+    *t1 = vtrn1q_s16(s1, s2);
+    *t2 = vtrn2q_s16(s1, s2);
 }
 
-static inline void transpose_4s_8h(int16x8_t &t1, int16x8_t &t2,
-                                   const int16x8_t s1, const int16x8_t s2)
+static inline void transpose_s16_s32x2(int16x8_t *t1, int16x8_t *t2,
+                                       const int16x8_t s1, const int16x8_t s2)
 {
     int32x4_t tmp1 = vreinterpretq_s32_s16(s1);
     int32x4_t tmp2 = vreinterpretq_s32_s16(s2);
 
-    t1 = vreinterpretq_s16_s32(vtrn1q_s32(tmp1, tmp2));
-    t2 = vreinterpretq_s16_s32(vtrn2q_s32(tmp1, tmp2));
+    *t1 = vreinterpretq_s16_s32(vtrn1q_s32(tmp1, tmp2));
+    *t2 = vreinterpretq_s16_s32(vtrn2q_s32(tmp1, tmp2));
 }
 
-static inline void transpose_2d_8h(int16x8_t &t1, int16x8_t &t2,
-                                   const int16x8_t s1, const int16x8_t s2)
+static inline void transpose_s16_s64x2(int16x8_t *t1, int16x8_t *t2,
+                                       const int16x8_t s1, const int16x8_t s2)
 {
     int64x2_t tmp1 = vreinterpretq_s64_s16(s1);
     int64x2_t tmp2 = vreinterpretq_s64_s16(s2);
 
-    t1 = vreinterpretq_s16_s64(vtrn1q_s64(tmp1, tmp2));
-    t2 = vreinterpretq_s16_s64(vtrn2q_s64(tmp1, tmp2));
+    *t1 = vreinterpretq_s16_s64(vtrn1q_s64(tmp1, tmp2));
+    *t2 = vreinterpretq_s16_s64(vtrn2q_s64(tmp1, tmp2));
 }
 
-static inline void SUMSUB_ABCD(int16x8_t &s1, int16x8_t &d1, int16x8_t &s2, int16x8_t &d2,
-                               int16x8_t a, int16x8_t  b, int16x8_t  c, int16x8_t  d)
+static inline uint16x8_t max_abs_s16(const int16x8_t a, const int16x8_t b)
 {
-    SUMSUB_AB(s1, d1, a, b);
-    SUMSUB_AB(s2, d2, c, d);
+    uint16x8_t abs0 = vreinterpretq_u16_s16(vabsq_s16(a));
+    uint16x8_t abs1 = vreinterpretq_u16_s16(vabsq_s16(b));
+
+    return vmaxq_u16(abs0, abs1);
 }
 
-static inline void HADAMARD4_V(int16x8_t &r1, int16x8_t &r2, int16x8_t &r3, int16x8_t &r4,
-                               int16x8_t &t1, int16x8_t &t2, int16x8_t &t3, int16x8_t &t4)
+#if X265_DEPTH == 12
+static inline void sumsubq_s32(int32x4_t *sum, int32x4_t *sub, const int32x4_t a, const int32x4_t b)
 {
-    SUMSUB_ABCD(t1, t2, t3, t4, r1, r2, r3, r4);
-    SUMSUB_ABCD(r1, r3, r2, r4, t1, t3, t2, t4);
+    *sum = vaddq_s32(a, b);
+    *sub = vsubq_s32(a, b);
 }
 
-
-static int _satd_4x8_8x4_end_neon(int16x8_t v0, int16x8_t v1, int16x8_t v2, int16x8_t v3)
-
+static inline void sumsublq_s16(int32x4_t *sum_lo, int32x4_t *sum_hi,
+                                int32x4_t *sub_lo, int32x4_t *sub_hi,
+                                const int16x8_t a, const int16x8_t b)
 {
+    *sum_lo = vaddl_s16(vget_low_s16(a), vget_low_s16(b));
+    *sub_lo = vsubl_s16(vget_low_s16(a), vget_low_s16(b));
+    *sum_hi = vaddl_s16(vget_high_s16(a), vget_high_s16(b));
+    *sub_hi = vsubl_s16(vget_high_s16(a), vget_high_s16(b));
+}
 
-    int16x8_t v4, v5, v6, v7, v16, v17, v18, v19;
-
-
-    SUMSUB_AB(v16, v17, v0,  v1);
-    SUMSUB_AB(v18, v19, v2,  v3);
-
-    SUMSUB_AB(v4 , v6 , v16, v18);
-    SUMSUB_AB(v5 , v7 , v17, v19);
-
-    transpose_8h_8h(v0, v1, v4, v5);
-    transpose_8h_8h(v2, v3, v6, v7);
+static inline void transpose_inplace_s32_s64x2(int32x4_t *t1, int32x4_t *t2)
+{
+    int64x2_t tmp1 = vreinterpretq_s64_s32(*t1);
+    int64x2_t tmp2 = vreinterpretq_s64_s32(*t2);
 
-    SUMSUB_AB(v16, v17, v0,  v1);
-    SUMSUB_AB(v18, v19, v2,  v3);
+    *t1 = vreinterpretq_s32_s64(vtrn1q_s64(tmp1, tmp2));
+    *t2 = vreinterpretq_s32_s64(vtrn2q_s64(tmp1, tmp2));
+}
 
-    transpose_4s_8h(v0, v1, v16, v18);
-    transpose_4s_8h(v2, v3, v17, v19);
+static inline uint32x4_t max_abs_s32(int32x4_t a, int32x4_t b)
+{
+    uint32x4_t abs0 = vreinterpretq_u32_s32(vabsq_s32(a));
+    uint32x4_t abs1 = vreinterpretq_u32_s32(vabsq_s32(b));
 
-    uint16x8_t abs0 = vreinterpretq_u16_s16(vabsq_s16(v0));
-    uint16x8_t abs1 = vreinterpretq_u16_s16(vabsq_s16(v1));
-    uint16x8_t abs2 = vreinterpretq_u16_s16(vabsq_s16(v2));
-    uint16x8_t abs3 = vreinterpretq_u16_s16(vabsq_s16(v3));
+    return vmaxq_u32(abs0, abs1);
+}
 
-    uint16x8_t max0 = vmaxq_u16(abs0, abs1);
-    uint16x8_t max1 = vmaxq_u16(abs2, abs3);
+#endif // X265_DEPTH == 12
 
-    uint16x8_t sum = vaddq_u16(max0, max1);
-    return vaddlvq_u16(sum);
+#if HIGH_BIT_DEPTH
+static inline void load_diff_u16x8x4(const uint16_t *pix1, intptr_t stride_pix1,
+                                     const uint16_t *pix2, intptr_t stride_pix2, int16x8_t diff4)
+{
+    uint16x8_t r4, t4;
+    load_u16x8xn<4>(pix1, stride_pix1, r);
+    load_u16x8xn<4>(pix2, stride_pix2, t);
+
+    diff0 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
+    diff1 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
+    diff2 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
+    diff3 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
 }
 
-static inline int _satd_4x4_neon(int16x8_t v0, int16x8_t v1)
+static inline void load_diff_u16x8x4_dual(const uint16_t *pix1, intptr_t stride_pix1,
+                                          const uint16_t *pix2, intptr_t stride_pix2, int16x8_t diff8)
 {
-    int16x8_t v2, v3;
-    SUMSUB_AB(v2,  v3,  v0,  v1);
-
-    transpose_2d_8h(v0, v1, v2, v3);
-    SUMSUB_AB(v2,  v3,  v0,  v1);
-
-    transpose_8h_8h(v0, v1, v2, v3);
-    SUMSUB_AB(v2,  v3,  v0,  v1);
+    load_diff_u16x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u16x8x4(pix1 + 4 * stride_pix1, stride_pix1,
+                      pix2 + 4 * stride_pix2, stride_pix2, diff + 4);
+}
 
-    transpose_4s_8h(v0, v1, v2, v3);
+static inline void load_diff_u16x8x8(const uint16_t *pix1, intptr_t stride_pix1,
+                                     const uint16_t *pix2, intptr_t stride_pix2, int16x8_t diff8)
+{
+    uint16x8_t r8, t8;
+    load_u16x8xn<8>(pix1, stride_pix1, r);
+    load_u16x8xn<8>(pix2, stride_pix2, t);
+
+    diff0 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
+    diff1 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
+    diff2 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
+    diff3 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
+    diff4 = vreinterpretq_s16_u16(vsubq_u16(r4, t4));
+    diff5 = vreinterpretq_s16_u16(vsubq_u16(r5, t5));
+    diff6 = vreinterpretq_s16_u16(vsubq_u16(r6, t6));
+    diff7 = vreinterpretq_s16_u16(vsubq_u16(r7, t7));
+}
 
-    uint16x8_t abs0 = vreinterpretq_u16_s16(vabsq_s16(v0));
-    uint16x8_t abs1 = vreinterpretq_u16_s16(vabsq_s16(v1));
-    uint16x8_t max = vmaxq_u16(abs0, abs1);
+#else // !HIGH_BIT_DEPTH
+static inline void load_diff_u8x8x4(const uint8_t *pix1, intptr_t stride_pix1,
+                                    const uint8_t *pix2, intptr_t stride_pix2, int16x8_t diff4)
+{
+    uint8x8_t r4, t4;
+    load_u8x8xn<4>(pix1, stride_pix1, r);
+    load_u8x8xn<4>(pix2, stride_pix2, t);
+
+    diff0 = vreinterpretq_s16_u16(vsubl_u8(r0, t0));
+    diff1 = vreinterpretq_s16_u16(vsubl_u8(r1, t1));
+    diff2 = vreinterpretq_s16_u16(vsubl_u8(r2, t2));
+    diff3 = vreinterpretq_s16_u16(vsubl_u8(r3, t3));
+}
 
-    return vaddlvq_u16(max);
+static inline void load_diff_u8x8x8(const uint8_t *pix1, intptr_t stride_pix1,
+                                    const uint8_t *pix2, intptr_t stride_pix2, int16x8_t diff8)
+{
+    load_diff_u8x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u8x8x4(pix1 + 4 * stride_pix1, stride_pix1,
+                     pix2 + 4 * stride_pix2, stride_pix2, diff + 4);
 }
 
-static void _satd_8x4v_8x8h_neon(int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3, int16x8_t &v20,
-                                 int16x8_t &v21, int16x8_t &v22, int16x8_t &v23)
+static inline void load_diff_u8x16x4(const uint8_t *pix1, intptr_t stride_pix1,
+                                     const uint8_t *pix2, intptr_t stride_pix2, int16x8_t diff8)
 {
-    int16x8_t v16, v17, v18, v19, v4, v5, v6, v7;
+    uint8x16_t s14, s24;
+    load_u8x16xn<4>(pix1, stride_pix1, s1);
+    load_u8x16xn<4>(pix2, stride_pix2, s2);
+
+    diff0 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s10), vget_low_u8(s20)));
+    diff1 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s11), vget_low_u8(s21)));
+    diff2 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s12), vget_low_u8(s22)));
+    diff3 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(s13), vget_low_u8(s23)));
+    diff4 = vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s10), vget_high_u8(s20)));
+    diff5 = vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s11), vget_high_u8(s21)));
+    diff6 = vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s12), vget_high_u8(s22)));
+    diff7 = vreinterpretq_s16_u16(vsubl_u8(vget_high_u8(s13), vget_high_u8(s23)));
+}
 
-    SUMSUB_AB(v16, v18, v0,  v2);
-    SUMSUB_AB(v17, v19, v1,  v3);
+#endif // HIGH_BIT_DEPTH
 
-    HADAMARD4_V(v20, v21, v22, v23, v0,  v1, v2, v3);
+// 4 way hadamard vertical pass.
+static inline void hadamard_4_v(const int16x8_t in_coefs4, int16x8_t out_coefs4)
+{
+    int16x8_t s0, s1, d0, d1;
 
-    transpose_8h_8h(v0,  v1,  v16, v17);
-    transpose_8h_8h(v2,  v3,  v18, v19);
-    transpose_8h_8h(v4,  v5,  v20, v21);
-    transpose_8h_8h(v6,  v7,  v22, v23);
+    sumsubq_s16(&s0, &d0, in_coefs0, in_coefs1);
+    sumsubq_s16(&s1, &d1, in_coefs2, in_coefs3);
 
-    SUMSUB_AB(v16, v17, v0,  v1);
-    SUMSUB_AB(v18, v19, v2,  v3);
-    SUMSUB_AB(v20, v21, v4,  v5);
-    SUMSUB_AB(v22, v23, v6,  v7);
+    sumsubq_s16(&out_coefs0, &out_coefs2, s0, s1);
+    sumsubq_s16(&out_coefs1, &out_coefs3, d0, d1);
+}
 
-    transpose_4s_8h(v0,  v2,  v16, v18);
-    transpose_4s_8h(v1,  v3,  v17, v19);
-    transpose_4s_8h(v4,  v6,  v20, v22);
-    transpose_4s_8h(v5,  v7,  v21, v23);
+// 8 way hadamard vertical pass.
+static inline void hadamard_8_v(const int16x8_t in_coefs8, int16x8_t out_coefs8)
+{
+    int16x8_t temp8;
 
-    uint16x8_t abs0 = vreinterpretq_u16_s16(vabsq_s16(v0));
-    uint16x8_t abs1 = vreinterpretq_u16_s16(vabsq_s16(v1));
-    uint16x8_t abs2 = vreinterpretq_u16_s16(vabsq_s16(v2));
-    uint16x8_t abs3 = vreinterpretq_u16_s16(vabsq_s16(v3));
-    uint16x8_t abs4 = vreinterpretq_u16_s16(vabsq_s16(v4));
-    uint16x8_t abs5 = vreinterpretq_u16_s16(vabsq_s16(v5));
-    uint16x8_t abs6 = vreinterpretq_u16_s16(vabsq_s16(v6));
-    uint16x8_t abs7 = vreinterpretq_u16_s16(vabsq_s16(v7));
+    hadamard_4_v(in_coefs, temp);
+    hadamard_4_v(in_coefs + 4, temp + 4);
 
-    v0 = vreinterpretq_s16_u16(vmaxq_u16(abs0, abs2));
-    v1 = vreinterpretq_s16_u16(vmaxq_u16(abs1, abs3));
-    v2 = vreinterpretq_s16_u16(vmaxq_u16(abs4, abs6));
-    v3 = vreinterpretq_s16_u16(vmaxq_u16(abs5, abs7));
+    sumsubq_s16(&out_coefs0, &out_coefs4, temp0, temp4);
+    sumsubq_s16(&out_coefs1, &out_coefs5, temp1, temp5);
+    sumsubq_s16(&out_coefs2, &out_coefs6, temp2, temp6);
+    sumsubq_s16(&out_coefs3, &out_coefs7, temp3, temp7);
 }
 
-#if HIGH_BIT_DEPTH
-
-#if (X265_DEPTH > 10)
-static inline void transpose_2d_4s(int32x4_t &t1, int32x4_t &t2,
-                                   const int32x4_t s1, const int32x4_t s2)
+// 4 way hadamard horizontal pass.
+static inline void hadamard_4_h(const int16x8_t in_coefs4, int16x8_t out_coefs4)
 {
-    int64x2_t tmp1 = vreinterpretq_s64_s32(s1);
-    int64x2_t tmp2 = vreinterpretq_s64_s32(s2);
+    int16x8_t s0, s1, d0, d1, t0, t1, t2, t3;
+
+    transpose_s16_s16x2(&t0, &t1, in_coefs0, in_coefs1);
+    transpose_s16_s16x2(&t2, &t3, in_coefs2, in_coefs3);
+
+    sumsubq_s16(&s0, &d0, t0, t1);
+    sumsubq_s16(&s1, &d1, t2, t3);
 
-    t1 = vreinterpretq_s32_s64(vtrn1q_s64(tmp1, tmp2));
-    t2 = vreinterpretq_s32_s64(vtrn2q_s64(tmp1, tmp2));
+    transpose_s16_s32x2(&out_coefs0, &out_coefs1, s0, s1);
+    transpose_s16_s32x2(&out_coefs2, &out_coefs3, d0, d1);
 }
 
-static inline void ISUMSUB_AB(int32x4_t &sum, int32x4_t &sub, const int32x4_t a, const int32x4_t b)
+#if X265_DEPTH != 12
+// 8 way hadamard horizontal pass.
+static inline void hadamard_8_h(int16x8_t coefs8, uint16x8_t out4)
 {
-    sum = vaddq_s32(a, b);
-    sub = vsubq_s32(a, b);
+    int16x8_t s0, s1, s2, s3, d0, d1, d2, d3;
+    int16x8_t temp8;
+
+    hadamard_4_h(coefs, temp);
+    hadamard_4_h(coefs + 4, temp + 4);
+
+    sumsubq_s16(&s0, &d0, temp0, temp1);
+    sumsubq_s16(&s1, &d1, temp2, temp3);
+    sumsubq_s16(&s2, &d2, temp4, temp5);
+    sumsubq_s16(&s3, &d3, temp6, temp7);
+
+    transpose_s16_s64x2(&temp0, &temp1, s0, s2);
+    transpose_s16_s64x2(&temp2, &temp3, s1, s3);
+    transpose_s16_s64x2(&temp4, &temp5, d0, d2);
+    transpose_s16_s64x2(&temp6, &temp7, d1, d3);
+
+    out0 = max_abs_s16(temp0, temp1);
+    out1 = max_abs_s16(temp2, temp3);
+    out2 = max_abs_s16(temp4, temp5);
+    out3 = max_abs_s16(temp6, temp7);
 }
 
-static inline void ISUMSUB_AB_FROM_INT16(int32x4_t &suml, int32x4_t &sumh, int32x4_t &subl, int32x4_t &subh,
-        const int16x8_t a, const int16x8_t b)
+#else // X265_DEPTH == 12
+static inline void hadamard_8_h(int16x8_t coefs8, uint32x4_t out4)
 {
-    suml = vaddl_s16(vget_low_s16(a), vget_low_s16(b));
-    sumh = vaddl_high_s16(a, b);
-    subl = vsubl_s16(vget_low_s16(a), vget_low_s16(b));
-    subh = vsubl_high_s16(a, b);
+    int16x8_t a8;
+
+    transpose_s16_s16x2(&a0, &a1, coefs0, coefs1);
+    transpose_s16_s16x2(&a2, &a3, coefs2, coefs3);
+    transpose_s16_s16x2(&a4, &a5, coefs4, coefs5);
+    transpose_s16_s16x2(&a6, &a7, coefs6, coefs7);
+
+    int32x4_t a_lo8, a_hi8, b_lo8, b_hi8;
+
+    sumsublq_s16(&a_lo0, &a_hi0, &a_lo4, &a_hi4, a0, a1);
+    sumsublq_s16(&a_lo1, &a_hi1, &a_lo5, &a_hi5, a2, a3);
+    sumsublq_s16(&a_lo2, &a_hi2, &a_lo6, &a_hi6, a4, a5);
+    sumsublq_s16(&a_lo3, &a_hi3, &a_lo7, &a_hi7, a6, a7);
+
+    transpose_inplace_s32_s64x2(&a_lo0, &a_lo1);
+    transpose_inplace_s32_s64x2(&a_lo2, &a_lo3);
+    transpose_inplace_s32_s64x2(&a_lo4, &a_lo5);
+    transpose_inplace_s32_s64x2(&a_lo6, &a_lo7);
+
+    transpose_inplace_s32_s64x2(&a_hi0, &a_hi1);
+    transpose_inplace_s32_s64x2(&a_hi2, &a_hi3);
+    transpose_inplace_s32_s64x2(&a_hi4, &a_hi5);
+    transpose_inplace_s32_s64x2(&a_hi6, &a_hi7);
+
+    sumsubq_s32(&b_lo0, &b_lo1, a_lo0, a_lo1);
+    sumsubq_s32(&b_lo2, &b_lo3, a_lo2, a_lo3);
+    sumsubq_s32(&b_lo4, &b_lo5, a_lo4, a_lo5);
+    sumsubq_s32(&b_lo6, &b_lo7, a_lo6, a_lo7);
+
+    sumsubq_s32(&b_hi0, &b_hi1, a_hi0, a_hi1);
+    sumsubq_s32(&b_hi2, &b_hi3, a_hi2, a_hi3);
+    sumsubq_s32(&b_hi4, &b_hi5, a_hi4, a_hi5);
+    sumsubq_s32(&b_hi6, &b_hi7, a_hi6, a_hi7);
+
+    uint32x4_t max0_lo = max_abs_s32(b_lo0, b_hi0);
+    uint32x4_t max1_lo = max_abs_s32(b_lo1, b_hi1);
+    uint32x4_t max2_lo = max_abs_s32(b_lo2, b_hi2);
+    uint32x4_t max3_lo = max_abs_s32(b_lo3, b_hi3);
+    uint32x4_t max0_hi = max_abs_s32(b_lo4, b_hi4);
+    uint32x4_t max1_hi = max_abs_s32(b_lo5, b_hi5);
+    uint32x4_t max2_hi = max_abs_s32(b_lo6, b_hi6);
+    uint32x4_t max3_hi = max_abs_s32(b_lo7, b_hi7);
+
+    out0 = vaddq_u32(max0_lo, max0_hi);
+    out1 = vaddq_u32(max1_lo, max1_hi);
+    out2 = vaddq_u32(max2_lo, max2_hi);
+    out3 = vaddq_u32(max3_lo, max3_hi);
 }
 
-#endif
+#endif // X265_DEPTH != 12
 
-static inline void _sub_8x8_fly(const uint16_t *pix1, intptr_t stride_pix1, const uint16_t *pix2, intptr_t stride_pix2,
-                                int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3,
-                                int16x8_t &v20, int16x8_t &v21, int16x8_t &v22, int16x8_t &v23)
+static inline int hadamard_4x4(int16x8_t a0, int16x8_t a1)
 {
-    uint16x8_t r0, r1, r2, r3;
-    uint16x8_t t0, t1, t2, t3;
-    int16x8_t v16, v17;
-    int16x8_t v18, v19;
-
-    r0 = vld1q_u16(pix1 + 0 * stride_pix1);
-    r1 = vld1q_u16(pix1 + 1 * stride_pix1);
-    r2 = vld1q_u16(pix1 + 2 * stride_pix1);
-    r3 = vld1q_u16(pix1 + 3 * stride_pix1);
-
-    t0 = vld1q_u16(pix2 + 0 * stride_pix2);
-    t1 = vld1q_u16(pix2 + 1 * stride_pix2);
-    t2 = vld1q_u16(pix2 + 2 * stride_pix2);
-    t3 = vld1q_u16(pix2 + 3 * stride_pix2);
-
-    v16 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
-    v17 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
-    v18 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
-    v19 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
+    int16x8_t sum, dif, t0, t1;
+    sumsubq_s16(&sum, &dif, a0, a1);
 
-    r0 = vld1q_u16(pix1 + 4 * stride_pix1);
-    r1 = vld1q_u16(pix1 + 5 * stride_pix1);
-    r2 = vld1q_u16(pix1 + 6 * stride_pix1);
-    r3 = vld1q_u16(pix1 + 7 * stride_pix1);
+    transpose_s16_s64x2(&t0, &t1, sum, dif);
+    sumsubq_s16(&sum, &dif, t0, t1);
 
-    t0 = vld1q_u16(pix2 + 4 * stride_pix2);
-    t1 = vld1q_u16(pix2 + 5 * stride_pix2);
-    t2 = vld1q_u16(pix2 + 6 * stride_pix2);
-    t3 = vld1q_u16(pix2 + 7 * stride_pix2);
+    transpose_s16_s16x2(&t0, &t1, sum, dif);
+    sumsubq_s16(&sum, &dif, t0, t1);
 
-    v20 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
-    v21 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
-    v22 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
-    v23 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
+    transpose_s16_s32x2(&t0, &t1, sum, dif);
 
-    SUMSUB_AB(v0,  v1,  v16, v17);
-    SUMSUB_AB(v2,  v3,  v18, v19);
+    uint16x8_t max = max_abs_s16(t0, t1);
 
+    return vaddlvq_u16(max);
 }
 
-
-
-
-static void _satd_16x4_neon(const uint16_t *pix1, intptr_t stride_pix1, const uint16_t *pix2, intptr_t stride_pix2,
-                            int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3)
+// Calculate 2 4x4 hadamard transformation.
+static void hadamard_4x4_dual(int16x8_t diff4, uint16x8_t *out)
 {
-    uint16x8_t r0, r1, r2, r3;
-    uint16x8_t t0, t1, t2, t3;
-    int16x8_t v16, v17, v20, v21;
-    int16x8_t v18, v19, v22, v23;
+    int16x8_t temp4;
 
-    r0 = vld1q_u16(pix1 + 0 * stride_pix1);
-    r1 = vld1q_u16(pix1 + 1 * stride_pix1);
-    r2 = vld1q_u16(pix1 + 2 * stride_pix1);
-    r3 = vld1q_u16(pix1 + 3 * stride_pix1);
+    hadamard_4_v(diff, temp);
+    hadamard_4_h(temp, diff);
 
-    t0 = vld1q_u16(pix2 + 0 * stride_pix2);
-    t1 = vld1q_u16(pix2 + 1 * stride_pix2);
-    t2 = vld1q_u16(pix2 + 2 * stride_pix2);
-    t3 = vld1q_u16(pix2 + 3 * stride_pix2);
+    uint16x8_t sum0 = max_abs_s16(diff0, diff1);
+    uint16x8_t sum1 = max_abs_s16(diff2, diff3);
 
-    v16 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
-    v17 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
-    v18 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
-    v19 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
-
-    r0 = vld1q_u16(pix1 + 0 * stride_pix1 + 8);
-    r1 = vld1q_u16(pix1 + 1 * stride_pix1 + 8);
-    r2 = vld1q_u16(pix1 + 2 * stride_pix1 + 8);
-    r3 = vld1q_u16(pix1 + 3 * stride_pix1 + 8);
+    *out = vaddq_u16(sum0, sum1);
+}
 
-    t0 = vld1q_u16(pix2 + 0 * stride_pix2 + 8);
-    t1 = vld1q_u16(pix2 + 1 * stride_pix2 + 8);
-    t2 = vld1q_u16(pix2 + 2 * stride_pix2 + 8);
-    t3 = vld1q_u16(pix2 + 3 * stride_pix2 + 8);
+// Calculate 4 4x4 hadamard transformation.
+static inline void hadamard_4x4_quad(int16x8_t diff8, uint16x8_t out2)
+{
+    int16x8_t temp8;
 
-    v20 = vreinterpretq_s16_u16(vsubq_u16(r0, t0));
-    v21 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
-    v22 = vreinterpretq_s16_u16(vsubq_u16(r2, t2));
-    v23 = vreinterpretq_s16_u16(vsubq_u16(r3, t3));
+    hadamard_4_v(diff, temp);
+    hadamard_4_v(diff + 4, temp + 4);
 
-    SUMSUB_AB(v0,  v1,  v16, v17);
-    SUMSUB_AB(v2,  v3,  v18, v19);
+    hadamard_4_h(temp, diff);
+    hadamard_4_h(temp + 4, diff + 4);
 
-    _satd_8x4v_8x8h_neon(v0, v1, v2, v3, v20, v21, v22, v23);
+    uint16x8_t sum0 = max_abs_s16(diff0, diff1);
+    uint16x8_t sum1 = max_abs_s16(diff2, diff3);
+    uint16x8_t sum2 = max_abs_s16(diff4, diff5);
+    uint16x8_t sum3 = max_abs_s16(diff6, diff7);
 
+    out0 = vaddq_u16(sum0, sum1);
+    out1 = vaddq_u16(sum2, sum3);
 }
 
-
-int pixel_satd_4x4_neon(const uint16_t *pix1, intptr_t stride_pix1, const uint16_t *pix2, intptr_t stride_pix2)
+#if X265_DEPTH == 8
+static inline void hadamard_8x8(int16x8_t diff8, uint16x8_t out2)
 {
-    uint16x4_t t0_0 = vld1_u16(pix1 + 0 * stride_pix1);
-    uint16x4_t t1_0 = vld1_u16(pix1 + 1 * stride_pix1);
-    uint16x4_t t0_1 = vld1_u16(pix1 + 2 * stride_pix1);
-    uint16x4_t t1_1 = vld1_u16(pix1 + 3 * stride_pix1);
-    uint16x8_t t0 = vcombine_u16(t0_0, t0_1);
-    uint16x8_t t1 = vcombine_u16(t1_0, t1_1);
+    int16x8_t temp8;
+    uint16x8_t sum4;
 
-    uint16x4_t r0_0 = vld1_u16(pix2 + 0 * stride_pix2);
-    uint16x4_t r1_0 = vld1_u16(pix2 + 1 * stride_pix2);
-    uint16x4_t r0_1 = vld1_u16(pix2 + 2 * stride_pix2);
-    uint16x4_t r1_1 = vld1_u16(pix2 + 3 * stride_pix2);
-    uint16x8_t r0 = vcombine_u16(r0_0, r0_1);
-    uint16x8_t r1 = vcombine_u16(r1_0, r1_1);
+    hadamard_8_v(diff, temp);
+    hadamard_8_h(temp, sum);
 
-    int16x8_t v0 = vreinterpretq_s16_u16(vsubq_u16(t0, r0));
-    int16x8_t v1 = vreinterpretq_s16_u16(vsubq_u16(r1, t1));
-
-    return _satd_4x4_neon(v0, v1);
+    out0 = vaddq_u16(sum0, sum1);
+    out1 = vaddq_u16(sum2, sum3);
 }
 
+#elif X265_DEPTH == 10
+static inline void hadamard_8x8(int16x8_t diff8, uint32x4_t out2)
+{
+    int16x8_t temp8;
+    uint16x8_t sum4;
 
+    hadamard_8_v(diff, temp);
+    hadamard_8_h(temp, sum);
 
+    out0 = vpaddlq_u16(sum0);
+    out1 = vpaddlq_u16(sum1);
+    out0 = vpadalq_u16(out0, sum2);
+    out1 = vpadalq_u16(out1, sum3);
+}
 
-
-
-int pixel_satd_8x4_neon(const uint16_t *pix1, intptr_t stride_pix1, const uint16_t *pix2, intptr_t stride_pix2)
+#elif X265_DEPTH == 12
+static inline void hadamard_8x8(int16x8_t diff8, uint32x4_t out2)
 {
-    uint16x8_t i0, i1, i2, i3, i4, i5, i6, i7;
-
-    i0 = vld1q_u16(pix1 + 0 * stride_pix1);
-    i1 = vld1q_u16(pix2 + 0 * stride_pix2);
-    i2 = vld1q_u16(pix1 + 1 * stride_pix1);
-    i3 = vld1q_u16(pix2 + 1 * stride_pix2);
-    i4 = vld1q_u16(pix1 + 2 * stride_pix1);
-    i5 = vld1q_u16(pix2 + 2 * stride_pix2);
-    i6 = vld1q_u16(pix1 + 3 * stride_pix1);
-    i7 = vld1q_u16(pix2 + 3 * stride_pix2);
+    int16x8_t temp8;
+    uint32x4_t sum4;
 
-    int16x8_t v0 = vreinterpretq_s16_u16(vsubq_u16(i0, i1));
-    int16x8_t v1 = vreinterpretq_s16_u16(vsubq_u16(i2, i3));
-    int16x8_t v2 = vreinterpretq_s16_u16(vsubq_u16(i4, i5));
-    int16x8_t v3 = vreinterpretq_s16_u16(vsubq_u16(i6, i7));
+    hadamard_8_v(diff, temp);
+    hadamard_8_h(temp, sum);
 
-    return _satd_4x8_8x4_end_neon(v0, v1, v2, v3);
+    out0 = vaddq_u32(sum0, sum1);
+    out1 = vaddq_u32(sum2, sum3);
 }
 
+#endif // X265_DEPTH == 8
 
-int pixel_satd_16x16_neon(const uint16_t *pix1, intptr_t stride_pix1, const uint16_t *pix2, intptr_t stride_pix2)
+#if HIGH_BIT_DEPTH
+static inline int pixel_satd_4x4_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                      const uint16_t *pix2, intptr_t stride_pix2)
 {
-    uint32x4_t v30 = vdupq_n_u32(0), v31 = vdupq_n_u32(0);
-    int16x8_t v0, v1, v2, v3;
+    uint16x4_t s4, r4;
+    load_u16x4xn<4>(pix1, stride_pix1, s);
+    load_u16x4xn<4>(pix2, stride_pix2, r);
 
-    for (int offset = 0; offset <= 12; offset += 4)
-    {
-        _satd_16x4_neon(pix1 + offset * stride_pix1, stride_pix1,
-                        pix2 + offset * stride_pix2,stride_pix2,
-                        v0, v1, v2, v3);
-        v30 = vpadalq_u16(v30, vreinterpretq_u16_s16(v0));
-        v30 = vpadalq_u16(v30, vreinterpretq_u16_s16(v1));
-        v31 = vpadalq_u16(v31, vreinterpretq_u16_s16(v2));
-        v31 = vpadalq_u16(v31, vreinterpretq_u16_s16(v3));
-    }
+    uint16x8_t s0 = vcombine_u16(s0, s2);
+    uint16x8_t s1 = vcombine_u16(s1, s3);
+    uint16x8_t r0 = vcombine_u16(r0, r2);
+    uint16x8_t r1 = vcombine_u16(r1, r3);
+
+    int16x8_t diff0 = vreinterpretq_s16_u16(vsubq_u16(s0, r0));
+    int16x8_t diff1 = vreinterpretq_s16_u16(vsubq_u16(r1, s1));
 
-    return vaddvq_u32(vaddq_u32(v30, v31));
+    return hadamard_4x4(diff0, diff1);
 }
 
-#else       //HIGH_BIT_DEPTH
+static inline int pixel_satd_4x8_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                      const uint16_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff4;
+
+    uint16x4_t s8, r8;
+    load_u16x4xn<8>(pix1, stride_pix1, s);
+    load_u16x4xn<8>(pix2, stride_pix2, r);
+
+    uint16x8_t s0 = vcombine_u16(s0, s4);
+    uint16x8_t s1 = vcombine_u16(s1, s5);
+    uint16x8_t s2 = vcombine_u16(s2, s6);
+    uint16x8_t s3 = vcombine_u16(s3, s7);
+    uint16x8_t r0 = vcombine_u16(r0, r4);
+    uint16x8_t r1 = vcombine_u16(r1, r5);
+    uint16x8_t r2 = vcombine_u16(r2, r6);
+    uint16x8_t r3 = vcombine_u16(r3, r7);
+
+    diff0 = vreinterpretq_s16_u16(vsubq_u16(s0, r0));
+    diff1 = vreinterpretq_s16_u16(vsubq_u16(r1, s1));
+    diff2 = vreinterpretq_s16_u16(vsubq_u16(s2, r2));
+    diff3 = vreinterpretq_s16_u16(vsubq_u16(r3, s3));
+
+    uint16x8_t out;
+    hadamard_4x4_dual(diff, &out);
+
+    return vaddlvq_u16(out);
+}
 
-static void _satd_16x4_neon(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2,
-                            int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3)
+static inline int pixel_satd_8x4_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                      const uint16_t *pix2, intptr_t stride_pix2)
 {
-    uint8x16_t r0, r1, r2, r3;
-    uint8x16_t t0, t1, t2, t3;
-    int16x8_t v16, v17, v20, v21;
-    int16x8_t v18, v19, v22, v23;
+    int16x8_t diff4;
+    load_diff_u16x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
 
-    r0 = vld1q_u8(pix1 + 0 * stride_pix1);
-    r1 = vld1q_u8(pix1 + 1 * stride_pix1);
-    r2 = vld1q_u8(pix1 + 2 * stride_pix1);
-    r3 = vld1q_u8(pix1 + 3 * stride_pix1);
+    uint16x8_t out;
+    hadamard_4x4_dual(diff, &out);
 
-    t0 = vld1q_u8(pix2 + 0 * stride_pix2);
-    t1 = vld1q_u8(pix2 + 1 * stride_pix2);
-    t2 = vld1q_u8(pix2 + 2 * stride_pix2);
-    t3 = vld1q_u8(pix2 + 3 * stride_pix2);
+    return vaddlvq_u16(out);
+}
 
-    v16 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(r0), vget_low_u8(t0)));
-    v20 = vreinterpretq_s16_u16(vsubl_high_u8(r0, t0));
-    v17 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(r1), vget_low_u8(t1)));
-    v21 = vreinterpretq_s16_u16(vsubl_high_u8(r1, t1));
-    v18 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(r2), vget_low_u8(t2)));
-    v22 = vreinterpretq_s16_u16(vsubl_high_u8(r2, t2));
-    v19 = vreinterpretq_s16_u16(vsubl_u8(vget_low_u8(r3), vget_low_u8(t3)));
-    v23 = vreinterpretq_s16_u16(vsubl_high_u8(r3, t3));
+static inline int pixel_satd_8x8_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                      const uint16_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff8;
+    uint16x8_t out2;
 
-    SUMSUB_AB(v0,  v1,  v16, v17);
-    SUMSUB_AB(v2,  v3,  v18, v19);
+    load_diff_u16x8x4_dual(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
 
-    _satd_8x4v_8x8h_neon(v0, v1, v2, v3, v20, v21, v22, v23);
+    uint32x4_t res = vpaddlq_u16(out0);
+    res = vpadalq_u16(res, out1);
 
+    return vaddvq_u32(res);
 }
 
-
-static inline void _sub_8x8_fly(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2,
-                                int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3,
-                                int16x8_t &v20, int16x8_t &v21, int16x8_t &v22, int16x8_t &v23)
+static inline int pixel_satd_8x16_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                       const uint16_t *pix2, intptr_t stride_pix2)
 {
-    uint8x8_t r0, r1, r2, r3;
-    uint8x8_t t0, t1, t2, t3;
-    int16x8_t v16, v17;
-    int16x8_t v18, v19;
+    int16x8_t diff16;
+    uint16x8_t out4;
+
+    load_diff_u16x8x4_dual(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u16x8x4_dual(pix1 + 8 * stride_pix1, stride_pix1,
+                           pix2 + 8 * stride_pix2, stride_pix2, diff + 8);
 
-    r0 = vld1_u8(pix1 + 0 * stride_pix1);
-    r1 = vld1_u8(pix1 + 1 * stride_pix1);
-    r2 = vld1_u8(pix1 + 2 * stride_pix1);
-    r3 = vld1_u8(pix1 + 3 * stride_pix1);
+    hadamard_4x4_quad(diff, out);
+    hadamard_4x4_quad(diff + 8, out + 2);
 
-    t0 = vld1_u8(pix2 + 0 * stride_pix2);
-    t1 = vld1_u8(pix2 + 1 * stride_pix2);
-    t2 = vld1_u8(pix2 + 2 * stride_pix2);
-    t3 = vld1_u8(pix2 + 3 * stride_pix2);
+    uint16x8_t sum0 = vaddq_u16(out0, out1);
+    uint16x8_t sum1 = vaddq_u16(out2, out3);
 
-    v16 = vreinterpretq_s16_u16(vsubl_u8(r0, t0));
-    v17 = vreinterpretq_s16_u16(vsubl_u8(r1, t1));
-    v18 = vreinterpretq_s16_u16(vsubl_u8(r2, t2));
-    v19 = vreinterpretq_s16_u16(vsubl_u8(r3, t3));
+    uint32x4_t res = vpaddlq_u16(sum0);
+    res = vpadalq_u16(res, sum1);
 
-    r0 = vld1_u8(pix1 + 4 * stride_pix1);
-    r1 = vld1_u8(pix1 + 5 * stride_pix1);
-    r2 = vld1_u8(pix1 + 6 * stride_pix1);
-    r3 = vld1_u8(pix1 + 7 * stride_pix1);
+    return vaddvq_u32(res);
+}
 
-    t0 = vld1_u8(pix2 + 4 * stride_pix2);
-    t1 = vld1_u8(pix2 + 5 * stride_pix2);
-    t2 = vld1_u8(pix2 + 6 * stride_pix2);
-    t3 = vld1_u8(pix2 + 7 * stride_pix2);
+static inline int pixel_satd_16x4_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                       const uint16_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff8;
 
-    v20 = vreinterpretq_s16_u16(vsubl_u8(r0, t0));
-    v21 = vreinterpretq_s16_u16(vsubl_u8(r1, t1));
-    v22 = vreinterpretq_s16_u16(vsubl_u8(r2, t2));
-    v23 = vreinterpretq_s16_u16(vsubl_u8(r3, t3));
+    load_diff_u16x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u16x8x4(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, diff + 4);
 
+    uint16x8_t sum0, sum1;
+    hadamard_4x4_dual(diff, &sum0);
+    hadamard_4x4_dual(diff + 4, &sum1);
 
-    SUMSUB_AB(v0,  v1,  v16, v17);
-    SUMSUB_AB(v2,  v3,  v18, v19);
+    sum0 = vaddq_u16(sum0, sum1);
 
+    return vaddlvq_u16(sum0);
 }
 
-int pixel_satd_4x4_neon(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2)
+static inline int pixel_satd_16x8_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                       const uint16_t *pix2, intptr_t stride_pix2)
 {
-    uint8x8_t t0 = load_u8x4x2(pix1, 2 * stride_pix1);
-    uint8x8_t t1 = load_u8x4x2(pix1 + stride_pix1, 2 * stride_pix1);
+    int16x8_t diff16;
+    uint16x8_t out4;
 
-    uint8x8_t r0 = load_u8x4x2(pix2, 2 * stride_pix2);
-    uint8x8_t r1 = load_u8x4x2(pix2 + stride_pix2, 2 * stride_pix2);
+    load_diff_u16x8x4_dual(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u16x8x4_dual(pix1 + 8, stride_pix1,  pix2 + 8, stride_pix2, diff + 8);
 
-    return _satd_4x4_neon(vreinterpretq_s16_u16(vsubl_u8(t0, r0)),
-                          vreinterpretq_s16_u16(vsubl_u8(r1, t1)));
-}
+    hadamard_4x4_quad(diff, out);
+    hadamard_4x4_quad(diff + 8, out + 2);
 
+#if X265_DEPTH == 10
+    uint16x8_t sum0 = vaddq_u16(out0, out1);
+    uint16x8_t sum1 = vaddq_u16(out2, out3);
 
-int pixel_satd_8x4_neon(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2)
-{
-    uint8x8_t i0, i1, i2, i3, i4, i5, i6, i7;
+    sum0 = vaddq_u16(sum0, sum1);
 
-    i0 = vld1_u8(pix1 + 0 * stride_pix1);
-    i1 = vld1_u8(pix2 + 0 * stride_pix2);
-    i2 = vld1_u8(pix1 + 1 * stride_pix1);
-    i3 = vld1_u8(pix2 + 1 * stride_pix2);
-    i4 = vld1_u8(pix1 + 2 * stride_pix1);
-    i5 = vld1_u8(pix2 + 2 * stride_pix2);
-    i6 = vld1_u8(pix1 + 3 * stride_pix1);
-    i7 = vld1_u8(pix2 + 3 * stride_pix2);
+    return vaddlvq_u16(sum0);
+#else // X265_DEPTH == 12
+    uint32x4_t sum0 = vpaddlq_u16(out0);
+    uint32x4_t sum1 = vpaddlq_u16(out1);
+    sum0 = vpadalq_u16(sum0, out2);
+    sum1 = vpadalq_u16(sum1, out3);
 
-    int16x8_t v0 = vreinterpretq_s16_u16(vsubl_u8(i0, i1));
-    int16x8_t v1 = vreinterpretq_s16_u16(vsubl_u8(i2, i3));
-    int16x8_t v2 = vreinterpretq_s16_u16(vsubl_u8(i4, i5));
-    int16x8_t v3 = vreinterpretq_s16_u16(vsubl_u8(i6, i7));
+    sum0 = vaddq_u32(sum0, sum1);
 
-    return _satd_4x8_8x4_end_neon(v0, v1, v2, v3);
+    return vaddvq_u32(sum0);
+#endif // X265_DEPTH == 10
 }
 
-int pixel_satd_16x16_neon(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2)
+static inline int pixel_satd_16x16_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                        const uint16_t *pix2, intptr_t stride_pix2)
 {
-    uint16x8_t v30, v31;
-    int16x8_t v0, v1, v2, v3;
-    uint16x8_t t0, t1;
+    uint32x4_t sum2= { vdupq_n_u32(0), vdupq_n_u32(0) };
+    int16x8_t diff8;
+    uint16x8_t out2;
 
-    _satd_16x4_neon(pix1, stride_pix1, pix2, stride_pix2, v0, v1, v2, v3);
-    v30 = vaddq_u16(vreinterpretq_u16_s16(v0), vreinterpretq_u16_s16(v1));
-    v31 = vaddq_u16(vreinterpretq_u16_s16(v2), vreinterpretq_u16_s16(v3));
+    for (int i = 0; i < 4; ++i)
+    {
+        load_diff_u16x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+        load_diff_u16x8x4(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, diff + 4);
 
-    _satd_16x4_neon(pix1 + 4 * stride_pix1, stride_pix1, pix2 + 4 * stride_pix2, stride_pix2, v0, v1, v2, v3);
-    t0 = vaddq_u16(vreinterpretq_u16_s16(v0), vreinterpretq_u16_s16(v1));
-    t1 = vaddq_u16(vreinterpretq_u16_s16(v2), vreinterpretq_u16_s16(v3));
-    v30 = vaddq_u16(v30, t0);
-    v31 = vaddq_u16(v31, t1);
+        hadamard_4x4_quad(diff, out);
 
-    _satd_16x4_neon(pix1 + 8 * stride_pix1, stride_pix1, pix2 + 8 * stride_pix2, stride_pix2, v0, v1, v2, v3);
-    t0 = vaddq_u16(vreinterpretq_u16_s16(v0), vreinterpretq_u16_s16(v1));
-    t1 = vaddq_u16(vreinterpretq_u16_s16(v2), vreinterpretq_u16_s16(v3));
-    v30 = vaddq_u16(v30, t0);
-    v31 = vaddq_u16(v31, t1);
+        sum0 = vpadalq_u16(sum0, out0);
+        sum1 = vpadalq_u16(sum1, out1);
 
-    _satd_16x4_neon(pix1 + 12 * stride_pix1, stride_pix1, pix2 + 12 * stride_pix2, stride_pix2, v0, v1, v2, v3);
-    t0 = vaddq_u16(vreinterpretq_u16_s16(v0), vreinterpretq_u16_s16(v1));
-    t1 = vaddq_u16(vreinterpretq_u16_s16(v2), vreinterpretq_u16_s16(v3));
-    v30 = vaddq_u16(v30, t0);
-    v31 = vaddq_u16(v31, t1);
+        pix1 += 4 * stride_pix1;
+        pix2 += 4 * stride_pix2;
+    }
 
-    uint32x4_t sum0 = vpaddlq_u16(v30);
-    uint32x4_t sum1 = vpaddlq_u16(v31);
-    sum0 = vaddq_u32(sum0, sum1);
-    return vaddvq_u32(sum0);
+    return vaddvq_u32(vaddq_u32(sum0, sum1));
 }
-#endif      //HIGH_BIT_DEPTH
 
-#if HIGH_BIT_DEPTH
-typedef uint32x4_t sa8d_out_type;
-#else
-typedef uint16x8_t sa8d_out_type;
-#endif
-
-static inline void _sa8d_8x8_neon_end(int16x8_t v0, int16x8_t v1, int16x8_t v2,
-                                      int16x8_t v3, int16x8_t v20,
-                                      int16x8_t v21, int16x8_t v22,
-                                      int16x8_t v23, sa8d_out_type &out0,
-                                      sa8d_out_type &out1)
+static inline int pixel_sa8d_8x8_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                      const uint16_t *pix2, intptr_t stride_pix2)
 {
-    int16x8_t v16, v17, v18, v19;
-    int16x8_t v4, v5, v6, v7;
+    int16x8_t diff8;
+    uint32x4_t res2;
 
-    SUMSUB_AB(v16, v18, v0,  v2);
-    SUMSUB_AB(v17, v19, v1,  v3);
+    load_diff_u16x8x4_dual(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
 
-    HADAMARD4_V(v20, v21, v22, v23, v0,  v1, v2, v3);
+    uint32x4_t s = vaddq_u32(res0, res1);
 
-    SUMSUB_AB(v0,  v16, v16, v20);
-    SUMSUB_AB(v1,  v17, v17, v21);
-    SUMSUB_AB(v2,  v18, v18, v22);
-    SUMSUB_AB(v3,  v19, v19, v23);
-
-    transpose_8h_8h(v20, v21, v16, v17);
-    transpose_8h_8h(v4,  v5,  v0,  v1);
-    transpose_8h_8h(v22, v23, v18, v19);
-    transpose_8h_8h(v6,  v7,  v2,  v3);
+    return (vaddvq_u32(s) + 1) >> 1;
+}
 
-#if (X265_DEPTH <= 10)
+static inline int pixel_sa8d_16x16_neon(const uint16_t *pix1, intptr_t stride_pix1,
+                                        const uint16_t *pix2, intptr_t stride_pix2)
+{
+    uint32x4_t sum0, sum1;
 
-    int16x8_t v24, v25;
+    int16x8_t diff8;
+    uint32x4_t res2;
 
-    SUMSUB_AB(v2,  v3,  v20, v21);
-    SUMSUB_AB(v24, v25, v4,  v5);
-    SUMSUB_AB(v0,  v1,  v22, v23);
-    SUMSUB_AB(v4,  v5,  v6,  v7);
+    load_diff_u16x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vaddq_u32(res0, res1);
 
-    transpose_4s_8h(v20, v22, v2,  v0);
-    transpose_4s_8h(v21, v23, v3,  v1);
-    transpose_4s_8h(v16, v18, v24, v4);
-    transpose_4s_8h(v17, v19, v25, v5);
+    load_diff_u16x8x8(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum1 = vaddq_u32(res0, res1);
 
-    SUMSUB_AB(v0,  v2,  v20, v22);
-    SUMSUB_AB(v1,  v3,  v21, v23);
-    SUMSUB_AB(v4,  v6,  v16, v18);
-    SUMSUB_AB(v5,  v7,  v17, v19);
+    load_diff_u16x8x8(pix1 + 8 * stride_pix1, stride_pix1,
+                      pix2 + 8 * stride_pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vaddq_u32(sum0, res0);
+    sum1 = vaddq_u32(sum1, res1);
 
-    transpose_2d_8h(v16, v20,  v0,  v4);
-    transpose_2d_8h(v17, v21,  v1,  v5);
-    transpose_2d_8h(v18, v22,  v2,  v6);
-    transpose_2d_8h(v19, v23,  v3,  v7);
+    load_diff_u16x8x8(pix1 + 8 * stride_pix1 + 8, stride_pix1,
+                      pix2 + 8 * stride_pix2 + 8, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vaddq_u32(sum0, res0);
+    sum1 = vaddq_u32(sum1, res1);
 
-    uint16x8_t abs0 = vreinterpretq_u16_s16(vabsq_s16(v16));
-    uint16x8_t abs1 = vreinterpretq_u16_s16(vabsq_s16(v17));
-    uint16x8_t abs2 = vreinterpretq_u16_s16(vabsq_s16(v18));
-    uint16x8_t abs3 = vreinterpretq_u16_s16(vabsq_s16(v19));
-    uint16x8_t abs4 = vreinterpretq_u16_s16(vabsq_s16(v20));
-    uint16x8_t abs5 = vreinterpretq_u16_s16(vabsq_s16(v21));
-    uint16x8_t abs6 = vreinterpretq_u16_s16(vabsq_s16(v22));
-    uint16x8_t abs7 = vreinterpretq_u16_s16(vabsq_s16(v23));
+    sum0 = vaddq_u32(sum0, sum1);
 
-    uint16x8_t max0 = vmaxq_u16(abs0, abs4);
-    uint16x8_t max1 = vmaxq_u16(abs1, abs5);
-    uint16x8_t max2 = vmaxq_u16(abs2, abs6);
-    uint16x8_t max3 = vmaxq_u16(abs3, abs7);
+    return (vaddvq_u32(sum0) + 1) >> 1;
+}
 
-#if HIGH_BIT_DEPTH
-    out0 = vpaddlq_u16(max0);
-    out1 = vpaddlq_u16(max1);
-    out0 = vpadalq_u16(out0, max2);
-    out1 = vpadalq_u16(out1, max3);
-
-#else //HIGH_BIT_DEPTH
-
-    out0 = vaddq_u16(max0, max1);
-    out1 = vaddq_u16(max2, max3);
-
-#endif //HIGH_BIT_DEPTH
-
-#else // HIGH_BIT_DEPTH 12 bit only, switching math to int32, each int16x8 is up-convreted to 2 int32x4 (low and high)
-
-    int32x4_t v2l, v2h, v3l, v3h, v24l, v24h, v25l, v25h, v0l, v0h, v1l, v1h;
-    int32x4_t v22l, v22h, v23l, v23h;
-    int32x4_t v4l, v4h, v5l, v5h;
-    int32x4_t v6l, v6h, v7l, v7h;
-    int32x4_t v16l, v16h, v17l, v17h;
-    int32x4_t v18l, v18h, v19l, v19h;
-    int32x4_t v20l, v20h, v21l, v21h;
-
-    ISUMSUB_AB_FROM_INT16(v2l, v2h, v3l, v3h, v20, v21);
-    ISUMSUB_AB_FROM_INT16(v24l, v24h, v25l, v25h, v4, v5);
-
-    v22l = vmovl_s16(vget_low_s16(v22));
-    v22h = vmovl_high_s16(v22);
-    v23l = vmovl_s16(vget_low_s16(v23));
-    v23h = vmovl_high_s16(v23);
-
-    ISUMSUB_AB(v0l,  v1l,  v22l, v23l);
-    ISUMSUB_AB(v0h,  v1h,  v22h, v23h);
-
-    v6l = vmovl_s16(vget_low_s16(v6));
-    v6h = vmovl_high_s16(v6);
-    v7l = vmovl_s16(vget_low_s16(v7));
-    v7h = vmovl_high_s16(v7);
-
-    ISUMSUB_AB(v4l,  v5l,  v6l,  v7l);
-    ISUMSUB_AB(v4h,  v5h,  v6h,  v7h);
-
-    transpose_2d_4s(v20l, v22l, v2l,  v0l);
-    transpose_2d_4s(v21l, v23l, v3l,  v1l);
-    transpose_2d_4s(v16l, v18l, v24l, v4l);
-    transpose_2d_4s(v17l, v19l, v25l, v5l);
-
-    transpose_2d_4s(v20h, v22h, v2h,  v0h);
-    transpose_2d_4s(v21h, v23h, v3h,  v1h);
-    transpose_2d_4s(v16h, v18h, v24h, v4h);
-    transpose_2d_4s(v17h, v19h, v25h, v5h);
-
-    ISUMSUB_AB(v0l,  v2l,  v20l, v22l);
-    ISUMSUB_AB(v1l,  v3l,  v21l, v23l);
-    ISUMSUB_AB(v4l,  v6l,  v16l, v18l);
-    ISUMSUB_AB(v5l,  v7l,  v17l, v19l);
-
-    ISUMSUB_AB(v0h,  v2h,  v20h, v22h);
-    ISUMSUB_AB(v1h,  v3h,  v21h, v23h);
-    ISUMSUB_AB(v4h,  v6h,  v16h, v18h);
-    ISUMSUB_AB(v5h,  v7h,  v17h, v19h);
-
-    v16l = v0l;
-    v16h = v4l;
-    v20l = v0h;
-    v20h = v4h;
-
-    v17l = v1l;
-    v17h = v5l;
-    v21l = v1h;
-    v21h = v5h;
-
-    v18l = v2l;
-    v18h = v6l;
-    v22l = v2h;
-    v22h = v6h;
-
-    v19l = v3l;
-    v19h = v7l;
-    v23l = v3h;
-    v23h = v7h;
-
-    uint32x4_t abs0_lo = vreinterpretq_u32_s32(vabsq_s32(v16l));
-    uint32x4_t abs1_lo = vreinterpretq_u32_s32(vabsq_s32(v17l));
-    uint32x4_t abs2_lo = vreinterpretq_u32_s32(vabsq_s32(v18l));
-    uint32x4_t abs3_lo = vreinterpretq_u32_s32(vabsq_s32(v19l));
-    uint32x4_t abs4_lo = vreinterpretq_u32_s32(vabsq_s32(v20l));
-    uint32x4_t abs5_lo = vreinterpretq_u32_s32(vabsq_s32(v21l));
-    uint32x4_t abs6_lo = vreinterpretq_u32_s32(vabsq_s32(v22l));
-    uint32x4_t abs7_lo = vreinterpretq_u32_s32(vabsq_s32(v23l));
-
-    uint32x4_t abs0_hi = vreinterpretq_u32_s32(vabsq_s32(v16h));
-    uint32x4_t abs1_hi = vreinterpretq_u32_s32(vabsq_s32(v17h));
-    uint32x4_t abs2_hi = vreinterpretq_u32_s32(vabsq_s32(v18h));
-    uint32x4_t abs3_hi = vreinterpretq_u32_s32(vabsq_s32(v19h));
-    uint32x4_t abs4_hi = vreinterpretq_u32_s32(vabsq_s32(v20h));
-    uint32x4_t abs5_hi = vreinterpretq_u32_s32(vabsq_s32(v21h));
-    uint32x4_t abs6_hi = vreinterpretq_u32_s32(vabsq_s32(v22h));
-    uint32x4_t abs7_hi = vreinterpretq_u32_s32(vabsq_s32(v23h));
-
-    uint32x4_t max0_lo = vmaxq_u32(abs0_lo, abs4_lo);
-    uint32x4_t max1_lo = vmaxq_u32(abs1_lo, abs5_lo);
-    uint32x4_t max2_lo = vmaxq_u32(abs2_lo, abs6_lo);
-    uint32x4_t max3_lo = vmaxq_u32(abs3_lo, abs7_lo);
-
-    uint32x4_t max0_hi = vmaxq_u32(abs0_hi, abs4_hi);
-    uint32x4_t max1_hi = vmaxq_u32(abs1_hi, abs5_hi);
-    uint32x4_t max2_hi = vmaxq_u32(abs2_hi, abs6_hi);
-    uint32x4_t max3_hi = vmaxq_u32(abs3_hi, abs7_hi);
-
-    uint32x4_t sum0 = vaddq_u32(max0_lo, max0_hi);
-    uint32x4_t sum1 = vaddq_u32(max1_lo, max1_hi);
-    uint32x4_t sum2 = vaddq_u32(max2_lo, max2_hi);
-    uint32x4_t sum3 = vaddq_u32(max3_lo, max3_hi);
-
-    out0 = vaddq_u32(sum0, sum1);
-    out1 = vaddq_u32(sum2, sum3);
+#else // !HIGH_BIT_DEPTH
+static inline int pixel_satd_4x4_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
+{
+    uint8x8_t s0 = load_u8x4x2(pix1, 2 * stride_pix1);
+    uint8x8_t s1 = load_u8x4x2(pix1 + stride_pix1, 2 * stride_pix1);
 
+    uint8x8_t r0 = load_u8x4x2(pix2, 2 * stride_pix2);
+    uint8x8_t r1 = load_u8x4x2(pix2 + stride_pix2, 2 * stride_pix2);
 
-#endif
+    int16x8_t diff0 = vreinterpretq_s16_u16(vsubl_u8(s0, r0));
+    int16x8_t diff1 = vreinterpretq_s16_u16(vsubl_u8(r1, s1));
 
+    return hadamard_4x4(diff0, diff1);
 }
 
-
-
-static inline void _satd_8x8_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2,
-                                  int16x8_t &v0, int16x8_t &v1, int16x8_t &v2, int16x8_t &v3)
+static inline int pixel_satd_4x8_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
 {
+    int16x8_t diff4;
+
+    uint8x8_t s0 = load_u8x4x2(pix1 + 0 * stride_pix1, 4 * stride_pix1);
+    uint8x8_t s1 = load_u8x4x2(pix1 + 1 * stride_pix1, 4 * stride_pix1);
+    uint8x8_t s2 = load_u8x4x2(pix1 + 2 * stride_pix1, 4 * stride_pix1);
+    uint8x8_t s3 = load_u8x4x2(pix1 + 3 * stride_pix1, 4 * stride_pix1);
+    uint8x8_t r0 = load_u8x4x2(pix2 + 0 * stride_pix2, 4 * stride_pix2);
+    uint8x8_t r1 = load_u8x4x2(pix2 + 1 * stride_pix2, 4 * stride_pix2);
+    uint8x8_t r2 = load_u8x4x2(pix2 + 2 * stride_pix2, 4 * stride_pix2);
+    uint8x8_t r3 = load_u8x4x2(pix2 + 3 * stride_pix2, 4 * stride_pix2);
+
+    diff0 = vreinterpretq_s16_u16(vsubl_u8(s0, r0));
+    diff1 = vreinterpretq_s16_u16(vsubl_u8(r1, s1));
+    diff2 = vreinterpretq_s16_u16(vsubl_u8(s2, r2));
+    diff3 = vreinterpretq_s16_u16(vsubl_u8(r3, s3));
+
+    uint16x8_t out;
+    hadamard_4x4_dual(diff, &out);
+
+    return vaddlvq_u16(out);
+}
 
-    int16x8_t v20, v21, v22, v23;
-    _sub_8x8_fly(pix1, stride_pix1, pix2, stride_pix2, v0, v1, v2, v3, v20, v21, v22, v23);
-    _satd_8x4v_8x8h_neon(v0, v1, v2, v3, v20, v21, v22, v23);
+static inline int pixel_satd_8x4_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff4;
 
-}
+    load_diff_u8x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
 
+    uint16x8_t out;
+    hadamard_4x4_dual(diff, &out);
 
+    return vaddlvq_u16(out);
+}
 
-int pixel_satd_8x8_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+static inline int pixel_satd_8x8_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
 {
-    int16x8_t v0, v1, v2, v3;
+    int16x8_t diff8;
+    uint16x8_t out2;
 
-    _satd_8x8_neon(pix1, stride_pix1, pix2, stride_pix2, v0, v1, v2, v3);
-    uint16x8_t v30 = vaddq_u16(vreinterpretq_u16_s16(v0), vreinterpretq_u16_s16(v1));
-    uint16x8_t v31 = vaddq_u16(vreinterpretq_u16_s16(v2), vreinterpretq_u16_s16(v3));
+    load_diff_u8x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
 
-#if !(HIGH_BIT_DEPTH)
-    uint16x8_t sum = vaddq_u16(v30, v31);
-    return vaddvq_u32(vpaddlq_u16(sum));
-#else
-    uint32x4_t sum = vpaddlq_u16(v30);
-    sum = vpadalq_u16(sum, v31);
-    return vaddvq_u32(sum);
-#endif
-}
+    out0 = vaddq_u16(out0, out1);
 
+    return vaddlvq_u16(out0);
+}
 
-int pixel_sa8d_8x8_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+static inline int pixel_satd_8x16_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                       const uint8_t *pix2, intptr_t stride_pix2)
 {
-    int16x8_t v0, v1, v2, v3;
-    int16x8_t v20, v21, v22, v23;
-    sa8d_out_type res0, res1;
+    int16x8_t diff16;
+    uint16x8_t out4;
 
-    _sub_8x8_fly(pix1, stride_pix1, pix2, stride_pix2, v0, v1, v2, v3, v20, v21, v22, v23);
-    _sa8d_8x8_neon_end(v0, v1, v2, v3, v20, v21, v22, v23, res0, res1);
-
-#if HIGH_BIT_DEPTH
-    uint32x4_t s = vaddq_u32(res0, res1);
-    return (vaddvq_u32(s) + 1) >> 1;
-#else
-    return (vaddlvq_u16(vaddq_u16(res0, res1)) + 1) >> 1;
-#endif
-}
+    load_diff_u8x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u8x8x8(pix1 + 8 * stride_pix1, stride_pix1,
+                     pix2 + 8 * stride_pix2, stride_pix2, diff + 8);
 
+    hadamard_4x4_quad(diff, out);
+    hadamard_4x4_quad(diff + 8, out + 2);
 
+    uint16x8_t sum0 = vaddq_u16(out0, out1);
+    uint16x8_t sum1 = vaddq_u16(out2, out3);
 
+    sum0 = vaddq_u16(sum0, sum1);
 
+    return vaddlvq_u16(sum0);
+}
 
-int pixel_sa8d_16x16_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+static inline int pixel_satd_16x4_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                       const uint8_t *pix2, intptr_t stride_pix2)
 {
-    int16x8_t v0, v1, v2, v3;
-    int16x8_t v20, v21, v22, v23;
-    sa8d_out_type res0, res1;
-    uint32x4_t v30, v31;
-
-    _sub_8x8_fly(pix1, stride_pix1, pix2, stride_pix2, v0, v1, v2, v3, v20, v21, v22, v23);
-    _sa8d_8x8_neon_end(v0, v1, v2, v3, v20, v21, v22, v23, res0, res1);
+    int16x8_t diff8;
 
-#if !(HIGH_BIT_DEPTH)
-    v30 = vpaddlq_u16(res0);
-    v31 = vpaddlq_u16(res1);
-#else
-    v30 = vaddq_u32(res0, res1);
-#endif
+    load_diff_u8x8x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u8x8x4(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, diff + 4);
 
-    _sub_8x8_fly(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, v0, v1, v2, v3, v20, v21, v22, v23);
-    _sa8d_8x8_neon_end(v0, v1, v2, v3, v20, v21, v22, v23, res0, res1);
+    uint16x8_t out2;
+    hadamard_4x4_dual(diff, &out0);
+    hadamard_4x4_dual(diff + 4, &out1);
 
-#if !(HIGH_BIT_DEPTH)
-    v30 = vpadalq_u16(v30, res0);
-    v31 = vpadalq_u16(v31, res1);
-#else
-    v31 = vaddq_u32(res0, res1);
-#endif
+    out0 = vaddq_u16(out0, out1);
 
+    return vaddlvq_u16(out0);
+}
 
-    _sub_8x8_fly(pix1 + 8 * stride_pix1, stride_pix1, pix2 + 8 * stride_pix2, stride_pix2, v0, v1, v2, v3, v20, v21, v22,
-                 v23);
-    _sa8d_8x8_neon_end(v0, v1, v2, v3, v20, v21, v22, v23, res0, res1);
+static inline int pixel_satd_16x8_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                       const uint8_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff16;
+    uint16x8_t out4;
 
-#if !(HIGH_BIT_DEPTH)
-    v30 = vpadalq_u16(v30, res0);
-    v31 = vpadalq_u16(v31, res1);
-#else
-    v30 = vaddq_u32(v30, res0);
-    v31 = vaddq_u32(v31, res1);
-#endif
+    load_diff_u8x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    load_diff_u8x8x8(pix1 + 8, stride_pix1,  pix2 + 8, stride_pix2, diff + 8);
 
-    _sub_8x8_fly(pix1 + 8 * stride_pix1 + 8, stride_pix1, pix2 + 8 * stride_pix2 + 8, stride_pix2, v0, v1, v2, v3, v20, v21,
-                 v22, v23);
-    _sa8d_8x8_neon_end(v0, v1, v2, v3, v20, v21, v22, v23, res0, res1);
+    hadamard_4x4_quad(diff, out);
+    hadamard_4x4_quad(diff + 8, out + 2);
 
-#if !(HIGH_BIT_DEPTH)
-    v30 = vpadalq_u16(v30, res0);
-    v31 = vpadalq_u16(v31, res1);
-#else
-    v30 = vaddq_u32(v30, res0);
-    v31 = vaddq_u32(v31, res1);
-#endif
+    uint16x8_t sum0 = vaddq_u16(out0, out1);
+    uint16x8_t sum1 = vaddq_u16(out2, out3);
 
-    v30 = vaddq_u32(v30, v31);
+    sum0 = vaddq_u16(sum0, sum1);
 
-    return (vaddvq_u32(v30) + 1) >> 1;
+    return vaddlvq_u16(sum0);
 }
 
+static inline int pixel_satd_16x16_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                        const uint8_t *pix2, intptr_t stride_pix2)
+{
+    uint16x8_t sum2, out2;
+    int16x8_t diff8;
+
+    load_diff_u8x16x4(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
+    sum0 = out0;
+    sum1 = out1;
+
+    load_diff_u8x16x4(pix1 + 4 * stride_pix1, stride_pix1,
+                      pix2 + 4 * stride_pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
+    sum0 = vaddq_u16(sum0, out0);
+    sum1 = vaddq_u16(sum1, out1);
+
+    load_diff_u8x16x4(pix1 + 8 * stride_pix1, stride_pix1,
+                      pix2 + 8 * stride_pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
+    sum0 = vaddq_u16(sum0, out0);
+    sum1 = vaddq_u16(sum1, out1);
+
+    load_diff_u8x16x4(pix1 + 12 * stride_pix1, stride_pix1,
+                      pix2 + 12 * stride_pix2, stride_pix2, diff);
+    hadamard_4x4_quad(diff, out);
+    sum0 = vaddq_u16(sum0, out0);
+    sum1 = vaddq_u16(sum1, out1);
+
+    uint32x4_t sum0 = vpaddlq_u16(sum0);
+    uint32x4_t sum1 = vpaddlq_u16(sum1);
 
+    sum0 = vaddq_u32(sum0, sum1);
 
+    return vaddvq_u32(sum0);
+}
 
+static inline int pixel_sa8d_8x8_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
+{
+    int16x8_t diff8;
+    uint16x8_t res2;
 
+    load_diff_u8x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
 
+    return (vaddlvq_u16(vaddq_u16(res0, res1)) + 1) >> 1;
+}
 
-
-template<int size>
-void blockfill_s_neon(int16_t *dst, intptr_t dstride, int16_t val)
+static inline int pixel_sa8d_16x16_neon(const uint8_t *pix1, intptr_t stride_pix1,
+                                        const uint8_t *pix2, intptr_t stride_pix2)
 {
-    for (int y = 0; y < size; y++)
-    {
-        int x = 0;
-        int16x8_t v = vdupq_n_s16(val);
-        for (; (x + 8) <= size; x += 8)
-        {
-            vst1q_s16(dst + y * dstride + x, v);
-        }
-        for (; x < size; x++)
-        {
-            dsty * dstride + x = val;
-        }
-    }
+    int16x8_t diff8;
+    uint16x8_t res2;
+    uint32x4_t sum0, sum1;
+
+    load_diff_u8x8x8(pix1, stride_pix1, pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vpaddlq_u16(res0);
+    sum1 = vpaddlq_u16(res1);
+
+    load_diff_u8x8x8(pix1 + 8, stride_pix1, pix2 + 8, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vpadalq_u16(sum0, res0);
+    sum1 = vpadalq_u16(sum1, res1);
+
+    load_diff_u8x8x8(pix1 + 8 * stride_pix1, stride_pix1,
+                     pix2 + 8 * stride_pix2, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vpadalq_u16(sum0, res0);
+    sum1 = vpadalq_u16(sum1, res1);
+
+    load_diff_u8x8x8(pix1 + 8 * stride_pix1 + 8, stride_pix1,
+                     pix2 + 8 * stride_pix2 + 8, stride_pix2, diff);
+    hadamard_8x8(diff, res);
+    sum0 = vpadalq_u16(sum0, res0);
+    sum1 = vpadalq_u16(sum1, res1);
+
+    sum0 = vaddq_u32(sum0, sum1);
+
+    return (vaddvq_u32(sum0) + 1) >> 1;
 }
 
+#endif // HIGH_BIT_DEPTH
+
 template<int lx, int ly>
 int sad_pp_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
 {
@@ -909,313 +897,194 @@
     return sum;
 }
 
-template<int lx, int ly>
-void sad_x3_neon(const pixel *pix1, const pixel *pix2, const pixel *pix3, const pixel *pix4, intptr_t frefstride,
-                 int32_t *res)
+template<int size>
+void blockfill_s_neon(int16_t *dst, intptr_t dstride, int16_t val)
 {
-    res0 = 0;
-    res1 = 0;
-    res2 = 0;
-    for (int y = 0; y < ly; y++)
+    for (int h = 0; h < size; h++)
     {
-        int x = 0;
-        uint16x8_t vsum16_0 = vdupq_n_u16(0);
-        uint16x8_t vsum16_1 = vdupq_n_u16(0);
-        uint16x8_t vsum16_2 = vdupq_n_u16(0);
-#if HIGH_BIT_DEPTH
-        for (; (x + 8) <= lx; x += 8)
+        for (int w = 0; w + 16 <= size; w += 16)
         {
-            uint16x8_t p1 = vld1q_u16(pix1 + x);
-            uint16x8_t p2 = vld1q_u16(pix2 + x);
-            uint16x8_t p3 = vld1q_u16(pix3 + x);
-            uint16x8_t p4 = vld1q_u16(pix4 + x);
-            vsum16_0 = vabaq_u16(vsum16_0, p1, p2);
-            vsum16_1 = vabaq_u16(vsum16_1, p1, p3);
-            vsum16_2 = vabaq_u16(vsum16_2, p1, p4);
+            vst1q_s16(dst + h * dstride + w, vdupq_n_s16(val));
+            vst1q_s16(dst + h * dstride + w + 8, vdupq_n_s16(val));
         }
-        if (lx & 4)
+        if (size == 8)
         {
-            uint16x4_t p1 = vld1_u16(pix1 + x);
-            uint16x4_t p2 = vld1_u16(pix2 + x);
-            uint16x4_t p3 = vld1_u16(pix3 + x);
-            uint16x4_t p4 = vld1_u16(pix4 + x);
-            res0 += vaddlv_u16(vaba_u16(vdup_n_u16(0), p1, p2));
-            res1 += vaddlv_u16(vaba_u16(vdup_n_u16(0), p1, p3));
-            res2 += vaddlv_u16(vaba_u16(vdup_n_u16(0), p1, p4));
-            x += 4;
+            vst1q_s16(dst + h * dstride, vdupq_n_s16(val));
         }
-        if (lx >= 4)
+        if (size == 4)
         {
-            res0 += vaddlvq_u16(vsum16_0);
-            res1 += vaddlvq_u16(vsum16_1);
-            res2 += vaddlvq_u16(vsum16_2);
+            vst1_s16(dst + h * dstride, vdup_n_s16(val));
         }
-#else
+    }
+}
 
-        for (; (x + 16) <= lx; x += 16)
-        {
-            uint8x16_t p1 = vld1q_u8(pix1 + x);
-            uint8x16_t p2 = vld1q_u8(pix2 + x);
-            uint8x16_t p3 = vld1q_u8(pix3 + x);
-            uint8x16_t p4 = vld1q_u8(pix4 + x);
-            vsum16_0 = vabal_u8(vsum16_0, vget_low_u8(p1), vget_low_u8(p2));
-            vsum16_0 = vabal_high_u8(vsum16_0, p1, p2);
-            vsum16_1 = vabal_u8(vsum16_1, vget_low_u8(p1), vget_low_u8(p3));
-            vsum16_1 = vabal_high_u8(vsum16_1, p1, p3);
-            vsum16_2 = vabal_u8(vsum16_2, vget_low_u8(p1), vget_low_u8(p4));
-            vsum16_2 = vabal_high_u8(vsum16_2, p1, p4);
-        }
-        if (lx & 8)
+#if !HIGH_BIT_DEPTH
+template<int width, int height>
+void blockcopy_ps_neon(int16_t *dst, intptr_t dst_stride, const pixel *src,
+                       intptr_t src_stride)
+{
+    for (int h = 0; h < height; h++)
+    {
+        int w = 0;
+        for (; w + 16 <= width; w += 16)
         {
-            uint8x8_t p1 = vld1_u8(pix1 + x);
-            uint8x8_t p2 = vld1_u8(pix2 + x);
-            uint8x8_t p3 = vld1_u8(pix3 + x);
-            uint8x8_t p4 = vld1_u8(pix4 + x);
-            vsum16_0 = vabal_u8(vsum16_0, p1, p2);
-            vsum16_1 = vabal_u8(vsum16_1, p1, p3);
-            vsum16_2 = vabal_u8(vsum16_2, p1, p4);
-            x += 8;
+            uint8x16_t s = vld1q_u8(src + w);
+            uint8x16x2_t t = vzipq_u8(s, vdupq_n_u8(0));
+            int16x8x2_t s_s16;
+            s_s16.val0 = vreinterpretq_s16_u8(t.val0);
+            s_s16.val1 = vreinterpretq_s16_u8(t.val1);
+            vst1q_s16_x2(dst + w, s_s16);
         }
-        if (lx & 4)
+        if (width & 8)
         {
-            uint8x8_t p1 = load_u8x4x1(pix1 + x);
-            uint8x8_t p2 = load_u8x4x1(pix2 + x);
-            uint8x8_t p3 = load_u8x4x1(pix3 + x);
-            uint8x8_t p4 = load_u8x4x1(pix4 + x);
-            vsum16_0 = vabal_u8(vsum16_0, p1, p2);
-            vsum16_1 = vabal_u8(vsum16_1, p1, p3);
-            vsum16_2 = vabal_u8(vsum16_2, p1, p4);
-            x += 4;
+            uint8x8_t s = vld1_u8(src + w);
+            uint16x8_t s_u16 = vmovl_u8(s);
+            vst1q_s16(dst + w, vreinterpretq_s16_u16(s_u16));
+            w += 8;
         }
-        if (lx >= 4)
+        if (width & 4)
         {
-            res0 += vaddvq_u16(vsum16_0);
-            res1 += vaddvq_u16(vsum16_1);
-            res2 += vaddvq_u16(vsum16_2);
+            uint8x8_t s = load_u8x4x1(src + w);
+            uint16x4_t s_u16 = vget_low_u16(vmovl_u8(s));
+            vst1_s16(dst + w, vreinterpret_s16_u16(s_u16));
         }
 
-#endif
-        if (lx & 3) for (; x < lx; x++)
-            {
-                res0 += abs(pix1x - pix2x);
-                res1 += abs(pix1x - pix3x);
-                res2 += abs(pix1x - pix4x);
-            }
-
-        pix1 += FENC_STRIDE;
-        pix2 += frefstride;
-        pix3 += frefstride;
-        pix4 += frefstride;
+        dst += dst_stride;
+        src += src_stride;
     }
 }
+#endif // !HIGH_BIT_DEPTH
 
-template<int lx, int ly>
-void sad_x4_neon(const pixel *pix1, const pixel *pix2, const pixel *pix3, const pixel *pix4, const pixel *pix5,
-                 intptr_t frefstride, int32_t *res)
+template<int width, int height>
+void blockcopy_pp_neon(pixel *dst, intptr_t dst_stride, const pixel *src,
+                       intptr_t src_stride)
 {
-    uint32x4_t result = vdupq_n_u32(0);
-
-    for (int y = 0; y < ly; y++)
+    for (int h = 0; h < height; h++)
     {
-        int x = 0;
-        uint16x8_t vsum16_0 = vdupq_n_u16(0);
-        uint16x8_t vsum16_1 = vdupq_n_u16(0);
-        uint16x8_t vsum16_2 = vdupq_n_u16(0);
-        uint16x8_t vsum16_3 = vdupq_n_u16(0);
+        int w = 0;
 #if HIGH_BIT_DEPTH
-        for (; (x + 16) <= lx; x += 16)
+        for (; w + 16 <= width; w += 16)
         {
-            uint16x8x2_t p1 = vld1q_u16_x2(&pix1x);
-            uint16x8x2_t p2 = vld1q_u16_x2(&pix2x);
-            uint16x8x2_t p3 = vld1q_u16_x2(&pix3x);
-            uint16x8x2_t p4 = vld1q_u16_x2(&pix4x);
-            uint16x8x2_t p5 = vld1q_u16_x2(&pix5x);
-            vsum16_0 = vabaq_u16(vsum16_0, p1.val0, p2.val0);
-            vsum16_1 = vabaq_u16(vsum16_1, p1.val0, p3.val0);
-            vsum16_2 = vabaq_u16(vsum16_2, p1.val0, p4.val0);
-            vsum16_3 = vabaq_u16(vsum16_3, p1.val0, p5.val0);
-            vsum16_0 = vabaq_u16(vsum16_0, p1.val1, p2.val1);
-            vsum16_1 = vabaq_u16(vsum16_1, p1.val1, p3.val1);
-            vsum16_2 = vabaq_u16(vsum16_2, p1.val1, p4.val1);
-            vsum16_3 = vabaq_u16(vsum16_3, p1.val1, p5.val1);
-        }
-        if (lx & 8)
-        {
-            uint16x8_t p1 = vld1q_u16(pix1 + x);
-            uint16x8_t p2 = vld1q_u16(pix2 + x);
-            uint16x8_t p3 = vld1q_u16(pix3 + x);
-            uint16x8_t p4 = vld1q_u16(pix4 + x);
-            uint16x8_t p5 = vld1q_u16(pix5 + x);
-            vsum16_0 = vabaq_u16(vsum16_0, p1, p2);
-            vsum16_1 = vabaq_u16(vsum16_1, p1, p3);
-            vsum16_2 = vabaq_u16(vsum16_2, p1, p4);
-            vsum16_3 = vabaq_u16(vsum16_3, p1, p5);
-            x += 8;
+            uint16x8_t s0_lo = vld1q_u16(src + w);
+            uint16x8_t s0_hi = vld1q_u16(src + w + 8);
+            vst1q_u16(dst + w, s0_lo);
+            vst1q_u16(dst + w + 8, s0_hi);
         }
-        if (lx & 4)
+        if (width & 8)
         {
-            /* This is equivalent to getting the absolute difference of pix1x with each of
-             * pix2 - pix5, then summing across the vector (4 values each) and adding the
-             * result to result. */
-            uint16x8_t p1 = vreinterpretq_u16_u64(
-                vld1q_dup_u64((uint64_t *)&pix1x));
-            uint16x8_t p2_3 = vcombine_u16(vld1_u16(pix2 + x),
-                                           vld1_u16(pix3 + x));
-            uint16x8_t p4_5 = vcombine_u16(vld1_u16(pix4 + x),
-                                           vld1_u16(pix5 + x));
-
-            uint16x8_t a = vabdq_u16(p1, p2_3);
-            uint16x8_t b = vabdq_u16(p1, p4_5);
-
-            result = vpadalq_u16(result, vpaddq_u16(a, b));
-            x += 4;
+            uint16x8_t s0 = vld1q_u16(src + w);
+            vst1q_u16(dst + w, s0);
+            w += 8;
         }
-        if (lx >= 4)
+        if (width & 4)
         {
-            /* This is equivalent to adding across each of the sum vectors and then adding
-             * to result. */
-            uint32x4_t sum01 = vpaddlq_u16(vpaddq_u16(vsum16_0, vsum16_1));
-            uint32x4_t sum23 = vpaddlq_u16(vpaddq_u16(vsum16_2, vsum16_3));
-            result = vaddq_u32(result, vpaddq_u32(sum01, sum23));
+            uint16x4_t s0 = vld1_u16(src + w);
+            vst1_u16(dst + w, s0);
+            w += 4;
         }
-
 #else
-
-        for (; (x + 16) <= lx; x += 16)
+        for (; w + 32 <= width; w += 32)
         {
-            uint8x16_t p1 = vld1q_u8(pix1 + x);
-            uint8x16_t p2 = vld1q_u8(pix2 + x);
-            uint8x16_t p3 = vld1q_u8(pix3 + x);
-            uint8x16_t p4 = vld1q_u8(pix4 + x);
-            uint8x16_t p5 = vld1q_u8(pix5 + x);
-            vsum16_0 = vabal_u8(vsum16_0, vget_low_u8(p1), vget_low_u8(p2));
-            vsum16_0 = vabal_high_u8(vsum16_0, p1, p2);
-            vsum16_1 = vabal_u8(vsum16_1, vget_low_u8(p1), vget_low_u8(p3));
-            vsum16_1 = vabal_high_u8(vsum16_1, p1, p3);
-            vsum16_2 = vabal_u8(vsum16_2, vget_low_u8(p1), vget_low_u8(p4));
-            vsum16_2 = vabal_high_u8(vsum16_2, p1, p4);
-            vsum16_3 = vabal_u8(vsum16_3, vget_low_u8(p1), vget_low_u8(p5));
-            vsum16_3 = vabal_high_u8(vsum16_3, p1, p5);
+            uint8x16_t s0_lo = vld1q_u8(src + w);
+            uint8x16_t s0_hi = vld1q_u8(src + w + 16);
+            vst1q_u8(dst + w, s0_lo);
+            vst1q_u8(dst + w + 16, s0_hi);
         }
-        if (lx & 8)
+        if (width & 16)
         {
-            uint8x8_t p1 = vld1_u8(pix1 + x);
-            uint8x8_t p2 = vld1_u8(pix2 + x);
-            uint8x8_t p3 = vld1_u8(pix3 + x);
-            uint8x8_t p4 = vld1_u8(pix4 + x);
-            uint8x8_t p5 = vld1_u8(pix5 + x);
-            vsum16_0 = vabal_u8(vsum16_0, p1, p2);
-            vsum16_1 = vabal_u8(vsum16_1, p1, p3);
-            vsum16_2 = vabal_u8(vsum16_2, p1, p4);
-            vsum16_3 = vabal_u8(vsum16_3, p1, p5);
-            x += 8;
+            uint8x16_t s0 = vld1q_u8(src + w);
+            vst1q_u8(dst + w, s0);
+            w += 16;
         }
-        if (lx & 4)
+        if (width & 8)
         {
-            uint8x16_t p1 = vreinterpretq_u8_u32(
-                vld1q_dup_u32((uint32_t *)&pix1x));
-
-            uint32x4_t p_x4 = vdupq_n_u32(0);
-            p_x4 = vld1q_lane_u32((uint32_t *)&pix2x, p_x4, 0);
-            p_x4 = vld1q_lane_u32((uint32_t *)&pix3x, p_x4, 1);
-            p_x4 = vld1q_lane_u32((uint32_t *)&pix4x, p_x4, 2);
-            p_x4 = vld1q_lane_u32((uint32_t *)&pix5x, p_x4, 3);
-
-            uint16x8_t sum = vabdl_u8(vget_low_u8(p1),
-                                      vget_low_u8(vreinterpretq_u8_u32(p_x4)));
-            uint16x8_t sum2 = vabdl_high_u8(p1, vreinterpretq_u8_u32(p_x4));
-
-            result = vpadalq_u16(result, vpaddq_u16(sum, sum2));
+            uint8x8_t s0 = vld1_u8(src + w);
+            vst1_u8(dst + w, s0);
+            w += 8;
         }
-        if (lx >= 4)
+        if (width & 4)
         {
-            result0 += vaddvq_u16(vsum16_0);
-            result1 += vaddvq_u16(vsum16_1);
-            result2 += vaddvq_u16(vsum16_2);
-            result3 += vaddvq_u16(vsum16_3);
+            uint8x8_t s0 = load_u8x4x1(src + w);
+            store_u8x4x1(dst + w, s0);
+            w += 4;
         }
-
 #endif
-        if (lx & 3) for (; x < lx; x++)
+        for (; w < width; w++)
         {
-            result0 += abs(pix1x - pix2x);
-            result1 += abs(pix1x - pix3x);
-            result2 += abs(pix1x - pix4x);
-            result3 += abs(pix1x - pix5x);
+            dstw = srcw;
         }
 
-        pix1 += FENC_STRIDE;
-        pix2 += frefstride;
-        pix3 += frefstride;
-        pix4 += frefstride;
-        pix5 += frefstride;
+        src += src_stride;
+        dst += dst_stride;
     }
-
-    vst1q_s32(res, vreinterpretq_s32_u32(result));
 }
 
-
-template<int lx, int ly, class T1, class T2>
-sse_t sse_neon(const T1 *pix1, intptr_t stride_pix1, const T2 *pix2, intptr_t stride_pix2)
+template<int width, int height>
+void blockcopy_ss_neon(int16_t *dst, intptr_t dst_stride, const int16_t *src,
+                       intptr_t src_stride)
 {
-    sse_t sum = 0;
-
-    int32x4_t vsum1 = vdupq_n_s32(0);
-    int32x4_t vsum2 = vdupq_n_s32(0);
-    for (int y = 0; y < ly; y++)
+    for (int h = 0; h < height; h++)
     {
-        int x = 0;
-        for (; (x + 8) <= lx; x += 8)
+        int w = 0;
+        for (; w + 16 <= width; w += 16)
         {
-            int16x8_t tmp;
-            if (sizeof(T1) == 2 && sizeof(T2) == 2)
-            {
-                // We have to cast to the 'real' type so that this block
-                // will compile for both low and high bitdepth.
-                uint16x8_t vpix1 = vld1q_u16((const uint16_t*)pix1 + x);
-                uint16x8_t vpix2 = vld1q_u16((const uint16_t*)pix2 + x);
-                tmp = vreinterpretq_s16_u16(vsubq_u16(vpix1, vpix2));
-            }
-            else if (sizeof(T1) == 1 && sizeof(T2) == 1)
-            {
-                // We have to cast to the 'real' type so that this block
-                // will compile for both low and high bitdepth.
-                uint8x8_t vpix1 = vld1_u8((const uint8_t*)pix1 + x);
-                uint8x8_t vpix2 = vld1_u8((const uint8_t*)pix2 + x);
-                tmp = vreinterpretq_s16_u16(vsubl_u8(vpix1, vpix2));
-            }
-            else
-            {
-                X265_CHECK(false, "unsupported sse");
-            }
-            vsum1 = vmlal_s16(vsum1, vget_low_s16(tmp), vget_low_s16(tmp));
-            vsum2 = vmlal_high_s16(vsum2, tmp, tmp);
+            int16x8_t a0 = vld1q_s16(src + w + 0);
+            int16x8_t a1 = vld1q_s16(src + w + 8);
+            vst1q_s16(dst + w + 0, a0);
+            vst1q_s16(dst + w + 8, a1);
         }
-        for (; x < lx; x++)
+        if (width & 8)
         {
-            int tmp = pix1x - pix2x;
-            sum += (tmp * tmp);
+            vst1q_s16(dst + w, vld1q_s16(src + w));
+            w += 8;
         }
-
-        if (sizeof(T1) == 2 && sizeof(T2) == 2)
+        if (width & 4)
         {
-            int32x4_t vsum = vaddq_s32(vsum1, vsum2);
-            sum += vaddvq_s32(vsum);
-            vsum1 = vsum2 = vdupq_n_s32(0);
+            vst1_s16(dst + w, vld1_s16(src + w));
         }
 
-        pix1 += stride_pix1;
-        pix2 += stride_pix2;
+        dst += dst_stride;
+        src += src_stride;
     }
+}
 
-    int32x4_t vsum = vaddq_s32(vsum1, vsum2);
+#if !HIGH_BIT_DEPTH
+template<int width, int height>
+void blockcopy_sp_neon(pixel *dst, intptr_t dst_stride, const int16_t *src,
+                       intptr_t src_stride)
+{
+    for (int h = 0; h < height; h++)
+    {
+        int w = 0;
+        for (; w + 16 <= width; w += 16) {
+            int16x8_t s0 = vld1q_s16(src + w + 0);
+            int16x8_t s1 = vld1q_s16(src + w + 8);
+            int8x16_t s01 = vcombine_s8(vmovn_s16(s0), vmovn_s16(s1));
+            vst1q_u8(dst + w, vreinterpretq_u8_s8(s01));
+        }
+        if (width & 8)
+        {
+            int16x8_t s0 = vld1q_s16(src + w);
+            int8x8_t s0_s8 = vmovn_s16(s0);
+            vst1_u8(dst + w, vreinterpret_u8_s8(s0_s8));
+            w += 8;
+        }
+        if (width & 4)
+        {
+            int16x4_t s0 = vld1_s16(src + w);
+            int8x8_t s0_s8 = vmovn_s16(vcombine_s16(s0, vdup_n_s16(0)));
+            store_u8x4x1(dst + w, vreinterpret_u8_s8(s0_s8));
+        }
 
-    return sum + vaddvq_s32(vsum);
+        dst += dst_stride;
+        src += src_stride;
+    }
 }
-
+#endif // !HIGH_BIT_DEPTH
 
 template<int bx, int by>
-void blockcopy_ps_neon(int16_t *a, intptr_t stridea, const pixel *b, intptr_t strideb)
+void pixel_sub_ps_neon(int16_t *a, intptr_t dstride, const pixel *b0, const pixel *b1, intptr_t sstride0,
+                       intptr_t sstride1)
 {
     for (int y = 0; y < by; y++)
     {
@@ -1223,172 +1092,423 @@
         for (; (x + 8) <= bx; x += 8)
         {
 #if HIGH_BIT_DEPTH
-            vst1q_s16(a + x, vreinterpretq_s16_u16(vld1q_u16(b + x)));
+            uint16x8_t diff = vsubq_u16(vld1q_u16(b0 + x), vld1q_u16(b1 + x));
+            vst1q_s16(a + x, vreinterpretq_s16_u16(diff));
 #else
-            int16x8_t in = vreinterpretq_s16_u16(vmovl_u8(vld1_u8(b + x)));
-            vst1q_s16(a + x, in);
+            uint16x8_t diff = vsubl_u8(vld1_u8(b0 + x), vld1_u8(b1 + x));
+            vst1q_s16(a + x, vreinterpretq_s16_u16(diff));
 #endif
         }
         for (; x < bx; x++)
         {
-            ax = (int16_t)bx;
+            ax = (int16_t)(b0x - b1x);
         }
 
-        a += stridea;
-        b += strideb;
+        b0 += sstride0;
+        b1 += sstride1;
+        a += dstride;
     }
 }
 
-
-template<int bx, int by>
-void blockcopy_pp_neon(pixel *a, intptr_t stridea, const pixel *b, intptr_t strideb)
+template<int width, int height>
+void pixel_add_ps_neon(pixel *dst, intptr_t dstride, const pixel *src0,
+                       const int16_t *src1, intptr_t sstride0, intptr_t sstride1)
 {
-    for (int y = 0; y < by; y++)
+    for (int h = 0; h < height; h++)
     {
-        int x = 0;
 #if HIGH_BIT_DEPTH
-        for (; (x + 8) <= bx; x += 8)
+        for (int w = 0; w + 16 <= width; w += 16)
         {
-            vst1q_u16(a + x, vld1q_u16(b + x));
+            uint16x8_t s0_lo = vld1q_u16(src0 + w);
+            uint16x8_t s0_hi = vld1q_u16(src0 + w + 8);
+            int16x8_t s1_lo = vld1q_s16(src1 + w);
+            int16x8_t s1_hi = vld1q_s16(src1 + w + 8);
+
+            uint16x8_t sum_lo = vsqaddq_u16(s0_lo, s1_lo);
+            uint16x8_t sum_hi = vsqaddq_u16(s0_hi, s1_hi);
+
+            sum_lo = vminq_u16(sum_lo, vdupq_n_u16((1 << X265_DEPTH) - 1));
+            sum_hi = vminq_u16(sum_hi, vdupq_n_u16((1 << X265_DEPTH) - 1));
+
+            vst1q_u16(dst + w, sum_lo);
+            vst1q_u16(dst + w + 8, sum_hi);
         }
-        if (bx & 4)
+        if (width == 8)
         {
-            vst1_u16(a + x, vld1_u16(b + x));
-            x += 4;
+            uint16x8_t s0 = vld1q_u16(src0);
+            int16x8_t s1 = vld1q_s16(src1);
+
+            uint16x8_t sum = vsqaddq_u16(s0, s1);
+            sum = vminq_u16(sum, vdupq_n_u16((1 << X265_DEPTH) - 1));
+
+            vst1q_u16(dst, sum);
         }
-#else
-        for (; (x + 16) <= bx; x += 16)
+        if (width == 4)
         {
-            vst1q_u8(a + x, vld1q_u8(b + x));
+            int16x4_t s1 = vld1_s16(src1);
+            uint16x4_t s0 = vld1_u16(src0);
+
+            uint16x4_t sum = vsqadd_u16(s0, s1);
+            sum = vmin_u16(sum, vdup_n_u16((1 << X265_DEPTH) - 1));
+
+            vst1_u16(dst, sum);
         }
-        if (bx & 8)
+#else // !HIGH_BIT_DEPTH
+        for (int w = 0; w + 16 <= width; w += 16)
         {
-            vst1_u8(a + x, vld1_u8(b + x));
-            x += 8;
+            uint8x16_t s0 = vld1q_u8(src0 + w);
+            int16x8_t s1_lo = vld1q_s16(src1 + w);
+            int16x8_t s1_hi = vld1q_s16(src1 + w + 8);
+
+            uint16x8_t sum_lo = vaddw_u8(vreinterpretq_u16_s16(s1_lo), vget_low_u8(s0));
+            uint16x8_t sum_hi = vaddw_u8(vreinterpretq_u16_s16(s1_hi), vget_high_u8(s0));
+            uint8x8_t d0_lo = vqmovun_s16(vreinterpretq_s16_u16(sum_lo));
+            uint8x8_t d0_hi = vqmovun_s16(vreinterpretq_s16_u16(sum_hi));
+
+            vst1_u8(dst + w, d0_lo);
+            vst1_u8(dst + w + 8, d0_hi);
         }
-        if (bx & 4)
+        if (width == 8)
         {
-            store_u8x4x1(a + x, load_u8x4x1(b + x));
-            x += 4;
+            uint8x8_t s0 = vld1_u8(src0);
+            int16x8_t s1 = vld1q_s16(src1);
+
+            uint16x8_t sum = vaddw_u8(vreinterpretq_u16_s16(s1), s0);
+            uint8x8_t d0 = vqmovun_s16(vreinterpretq_s16_u16(sum));
+
+            vst1_u8(dst, d0);
         }
-#endif
-        for (; x < bx; x++)
+        if (width == 4)
         {
-            ax = bx;
+            uint8x8_t s0 = load_u8x4x1(src0);
+            int16x8_t s1 = vcombine_s16(vld1_s16(src1), vdup_n_s16(0));
+
+            uint16x8_t sum = vaddw_u8(vreinterpretq_u16_s16(s1), s0);
+            uint8x8_t d0 = vqmovun_s16(vreinterpretq_s16_u16(sum));
+
+            store_u8x4x1(dst, d0);
         }
+#endif
 
-        a += stridea;
-        b += strideb;
+        src0 += sstride0;
+        src1 += sstride1;
+        dst += dstride;
     }
 }
 
-
-template<int bx, int by>
-void pixel_sub_ps_neon(int16_t *a, intptr_t dstride, const pixel *b0, const pixel *b1, intptr_t sstride0,
-                       intptr_t sstride1)
+template<int width, int height>
+void addAvg_neon(const int16_t *src0, const int16_t *src1, pixel *dst,
+                 intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)
 {
-    for (int y = 0; y < by; y++)
+    const int shiftNum = IF_INTERNAL_PREC + 1 - X265_DEPTH;
+    const int offset = 2 * IF_INTERNAL_OFFS;
+
+#if HIGH_BIT_DEPTH
+    const int16x8_t addon = vdupq_n_s16(offset >> shiftNum);
+
+    for (int h = 0; h < height; h++)
     {
-        int x = 0;
-        for (; (x + 8) <= bx; x += 8)
+        int w = 0;
+        for (; w + 16 <= width; w += 16)
         {
-#if HIGH_BIT_DEPTH
-            uint16x8_t diff = vsubq_u16(vld1q_u16(b0 + x), vld1q_u16(b1 + x));
-            vst1q_s16(a + x, vreinterpretq_s16_u16(diff));
-#else
-            uint16x8_t diff = vsubl_u8(vld1_u8(b0 + x), vld1_u8(b1 + x));
-            vst1q_s16(a + x, vreinterpretq_s16_u16(diff));
-#endif
+            int16x8_t s02, s12;
+            load_s16x8xn<2>(src0 + w, 8, s0);
+            load_s16x8xn<2>(src1 + w, 8, s1);
+
+            int16x8_t d0_lo = vrsraq_n_s16(addon, vaddq_s16(s00, s10), shiftNum);
+            int16x8_t d0_hi = vrsraq_n_s16(addon, vaddq_s16(s01, s11), shiftNum);
+
+            d0_lo = vminq_s16(d0_lo, vdupq_n_s16((1 << X265_DEPTH) - 1));
+            d0_lo = vmaxq_s16(d0_lo, vdupq_n_s16(0));
+            d0_hi = vminq_s16(d0_hi, vdupq_n_s16((1 << X265_DEPTH) - 1));
+            d0_hi = vmaxq_s16(d0_hi, vdupq_n_s16(0));
+
+            vst1q_u16(dst + w, vreinterpretq_u16_s16(d0_lo));
+            vst1q_u16(dst + w + 8, vreinterpretq_u16_s16(d0_hi));
         }
-        for (; x < bx; x++)
+        if (width & 8)
         {
-            ax = (int16_t)(b0x - b1x);
+            int16x8_t s0 = vld1q_s16(src0 + w);
+            int16x8_t s1 = vld1q_s16(src1 + w);
+
+            int16x8_t d0 = vrsraq_n_s16(addon, vaddq_s16(s0, s1), shiftNum);
+            d0 = vminq_s16(d0, vdupq_n_s16((1 << X265_DEPTH) - 1));
+            d0 = vmaxq_s16(d0, vdupq_n_s16(0));
+
+            vst1q_u16(dst + w, vreinterpretq_u16_s16(d0));
+
+            w += 8;
         }
+        if (width & 4)
+        {
+            int16x4_t s0 = vld1_s16(src0 + w);
+            int16x4_t s1 = vld1_s16(src1 + w);
 
-        b0 += sstride0;
-        b1 += sstride1;
-        a += dstride;
+            int16x4_t d0 = vrsra_n_s16(vget_low_s16(addon), vadd_s16(s0, s1), shiftNum);
+            d0 = vmin_s16(d0, vdup_n_s16((1 << X265_DEPTH) - 1));
+            d0 = vmax_s16(d0, vdup_n_s16(0));
+
+            vst1_u16(dst + w, vreinterpret_u16_s16(d0));
+
+            w += 4;
+        }
+        if (width & 2)
+        {
+            int16x8_t s0 = load_s16x2x1(src0 + w);
+            int16x8_t s1 = load_s16x2x1(src1 + w);
+
+            int16x8_t d0 = vrsraq_n_s16(addon, vaddq_s16(s0, s1), shiftNum);
+            d0 = vminq_s16(d0, vdupq_n_s16((1 << X265_DEPTH) - 1));
+            d0 = vmaxq_s16(d0, vdupq_n_s16(0));
+
+            store_u16x2x1(dst + w, vreinterpretq_u16_s16(d0));
+        }
+
+        src0 += src0Stride;
+        src1 += src1Stride;
+        dst  += dstStride;
     }
-}
+#else // !HIGH_BIT_DEPTH
+    const uint8x8_t addon = vdup_n_u8(offset >> shiftNum);
 
-template<int bx, int by>
-void pixel_add_ps_neon(pixel *a, intptr_t dstride, const pixel *b0, const int16_t *b1, intptr_t sstride0,
-                       intptr_t sstride1)
-{
-    for (int y = 0; y < by; y++)
+    for (int h = 0; h < height; h++)
     {
-        int x = 0;
-        for (; (x + 8) <= bx; x += 8)
+        int w = 0;
+        for (; w + 16 <= width; w += 16)
         {
-            int16x8_t t;
-            int16x8_t b1e = vld1q_s16(b1 + x);
-            int16x8_t b0e;
-#if HIGH_BIT_DEPTH
-            b0e = vreinterpretq_s16_u16(vld1q_u16(b0 + x));
-            t = vaddq_s16(b0e, b1e);
-            t = vminq_s16(t, vdupq_n_s16((1 << X265_DEPTH) - 1));
-            t = vmaxq_s16(t, vdupq_n_s16(0));
-            vst1q_u16(a + x, vreinterpretq_u16_s16(t));
-#else
-            b0e = vreinterpretq_s16_u16(vmovl_u8(vld1_u8(b0 + x)));
-            t = vaddq_s16(b0e, b1e);
-            vst1_u8(a + x, vqmovun_s16(t));
-#endif
+            int16x8_t s02, s12;
+            load_s16x8xn<2>(src0 + w, 8, s0);
+            load_s16x8xn<2>(src1 + w, 8, s1);
+
+            int8x8_t sum01_s8_lo = vqrshrn_n_s16(vaddq_s16(s00, s10), shiftNum);
+            int8x8_t sum01_s8_hi = vqrshrn_n_s16(vaddq_s16(s01, s11), shiftNum);
+            uint8x8_t d0_lo = vadd_u8(vreinterpret_u8_s8(sum01_s8_lo), addon);
+            uint8x8_t d0_hi = vadd_u8(vreinterpret_u8_s8(sum01_s8_hi), addon);
+
+            vst1_u8(dst + w, d0_lo);
+            vst1_u8(dst + w + 8, d0_hi);
         }
-        for (; x < bx; x++)
+        if (width & 8)
         {
-            ax = (int16_t)x265_clip(b0x + b1x);
+            int16x8_t s0 = vld1q_s16(src0 + w);
+            int16x8_t s1 = vld1q_s16(src1 + w);
+
+            int8x8_t sum01_s8 = vqrshrn_n_s16(vaddq_s16(s0, s1), shiftNum);
+            uint8x8_t d0 = vadd_u8(vreinterpret_u8_s8(sum01_s8), addon);
+
+            vst1_u8(dst + w, d0);
+
+            w += 8;
         }
+        if (width & 4)
+        {
+            int16x8_t s0 = vcombine_s16(vld1_s16(src0 + w), vdup_n_s16(0));
+            int16x8_t s1 = vcombine_s16(vld1_s16(src1 + w), vdup_n_s16(0));
 
-        b0 += sstride0;
-        b1 += sstride1;
-        a += dstride;
+            int8x8_t sum01_s8 = vqrshrn_n_s16(vaddq_s16(s0, s1), shiftNum);
+            uint8x8_t d0 = vadd_u8(vreinterpret_u8_s8(sum01_s8), addon);
+
+            store_u8x4x1(dst + w, d0);
+
+            w += 4;
+        }
+        if (width & 2)
+        {
+            int16x8_t s0 = load_s16x2x1(src0 + w);
+            int16x8_t s1 = load_s16x2x1(src1 + w);
+
+            int8x8_t sum01_s8 = vqrshrn_n_s16(vaddq_s16(s0, s1), shiftNum);
+            uint8x8_t d0 = vadd_u8(vreinterpret_u8_s8(sum01_s8), addon);
+
+            store_u8x2x1(dst + w, d0);
+        }
+
+        src0 += src0Stride;
+        src1 += src1Stride;
+        dst  += dstStride;
     }
+#endif
 }
 
-template<int bx, int by>
-void addAvg_neon(const int16_t *src0, const int16_t *src1, pixel *dst, intptr_t src0Stride, intptr_t src1Stride,
-                 intptr_t dstStride)
+void planecopy_cp_neon(const uint8_t *src, intptr_t srcStride, pixel *dst,
+                       intptr_t dstStride, int width, int height, int shift)
 {
+    X265_CHECK(width >= 16, "width length error\n");
+    X265_CHECK(height >= 1, "height length error\n");
+    X265_CHECK(shift == X265_DEPTH - 8, "shift value error\n");
 
-    const int shiftNum = IF_INTERNAL_PREC + 1 - X265_DEPTH;
-    const int offset = (1 << (shiftNum - 1)) + 2 * IF_INTERNAL_OFFS;
+    (void)shift;
 
-    const int32x4_t addon = vdupq_n_s32(offset);
-    for (int y = 0; y < by; y++)
+    do
     {
-        int x = 0;
-
-        for (; (x + 8) <= bx; x += 8)
-        {
-            int16x8_t in0 = vld1q_s16(src0 + x);
-            int16x8_t in1 = vld1q_s16(src1 + x);
-            int32x4_t t1 = vaddl_s16(vget_low_s16(in0), vget_low_s16(in1));
-            int32x4_t t2 = vaddl_high_s16(in0, in1);
-            t1 = vaddq_s32(t1, addon);
-            t2 = vaddq_s32(t2, addon);
-            t1 = vshrq_n_s32(t1, shiftNum);
-            t2 = vshrq_n_s32(t2, shiftNum);
-            int16x8_t t = vuzp1q_s16(vreinterpretq_s16_s32(t1),
-                                     vreinterpretq_s16_s32(t2));
 #if HIGH_BIT_DEPTH
-            t = vminq_s16(t, vdupq_n_s16((1 << X265_DEPTH) - 1));
-            t = vmaxq_s16(t, vdupq_n_s16(0));
-            vst1q_u16(dst + x, vreinterpretq_u16_s16(t));
+        for (int w = 0; w < width - 16; w += 16)
+        {
+            uint8x16_t in = vld1q_u8(src + w);
+            uint16x8_t t0 = vshll_n_u8(vget_low_u8(in), X265_DEPTH - 8);
+            uint16x8_t t1 = vshll_n_u8(vget_high_u8(in), X265_DEPTH - 8);
+            vst1q_u16(dst + w + 0, t0);
+            vst1q_u16(dst + w + 8, t1);
+        }
+        // Tail - src must be different from dst for this to work.
+        {
+            uint8x16_t in = vld1q_u8(src + width - 16);
+            uint16x8_t t0 = vshll_n_u8(vget_low_u8(in), X265_DEPTH - 8);
+            uint16x8_t t1 = vshll_n_u8(vget_high_u8(in), X265_DEPTH - 8);
+            vst1q_u16(dst + width - 16, t0);
+            vst1q_u16(dst + width - 8, t1);
+        }
 #else
-            vst1_u8(dst + x, vqmovun_s16(t));
+        int w;
+        for (w = 0; w < width - 32; w += 32)
+        {
+            uint8x16_t in0 = vld1q_u8(src + w + 0);
+            uint8x16_t in1 = vld1q_u8(src + w + 16);
+            vst1q_u8(dst + w + 0, in0);
+            vst1q_u8(dst + w + 16, in1);
+        }
+        if (w < width - 16)
+        {
+            uint8x16_t in = vld1q_u8(src + w);
+            vst1q_u8(dst + w, in);
+        }
+        // Tail - src must be different from dst for this to work.
+        {
+            uint8x16_t in = vld1q_u8(src + width - 16);
+            vst1q_u8(dst + width - 16, in);
+        }
 #endif
+        dst += dstStride;
+        src += srcStride;
+    }
+    while (--height != 0);
+}
+
+void weight_pp_neon(const pixel *src, pixel *dst, intptr_t stride, int width, int height,
+                    int w0, int round, int shift, int offset)
+{
+    const int correction = IF_INTERNAL_PREC - X265_DEPTH;
+
+    X265_CHECK(height >= 1, "height length error\n");
+    X265_CHECK(width >= 16, "width length error\n");
+    X265_CHECK(!(width & 15), "width alignment error\n");
+    X265_CHECK(w0 >= 0, "w0 should be min 0\n");
+    X265_CHECK(w0 < 128, "w0 should be max 127\n");
+    X265_CHECK(shift >= correction, "shift must include factor correction\n");
+    X265_CHECK((round & ((1 << correction) - 1)) == 0,
+               "round must include factor correction\n");
+
+    (void)round;
+
+#if HIGH_BIT_DEPTH
+    int32x4_t corrected_shift = vdupq_n_s32(correction - shift);
+
+    do
+    {
+        int w = 0;
+        do
+        {
+            int16x8_t s0 = vreinterpretq_s16_u16(vld1q_u16(src + w + 0));
+            int16x8_t s1 = vreinterpretq_s16_u16(vld1q_u16(src + w + 8));
+            int32x4_t weighted_s0_lo = vmull_n_s16(vget_low_s16(s0), w0);
+            int32x4_t weighted_s0_hi = vmull_n_s16(vget_high_s16(s0), w0);
+            int32x4_t weighted_s1_lo = vmull_n_s16(vget_low_s16(s1), w0);
+            int32x4_t weighted_s1_hi = vmull_n_s16(vget_high_s16(s1), w0);
+            weighted_s0_lo = vrshlq_s32(weighted_s0_lo, corrected_shift);
+            weighted_s0_hi = vrshlq_s32(weighted_s0_hi, corrected_shift);
+            weighted_s1_lo = vrshlq_s32(weighted_s1_lo, corrected_shift);
+            weighted_s1_hi = vrshlq_s32(weighted_s1_hi, corrected_shift);
+            weighted_s0_lo = vaddq_s32(weighted_s0_lo, vdupq_n_s32(offset));
+            weighted_s0_hi = vaddq_s32(weighted_s0_hi, vdupq_n_s32(offset));
+            weighted_s1_lo = vaddq_s32(weighted_s1_lo, vdupq_n_s32(offset));
+            weighted_s1_hi = vaddq_s32(weighted_s1_hi, vdupq_n_s32(offset));
+            uint16x4_t t0_lo = vqmovun_s32(weighted_s0_lo);
+            uint16x4_t t0_hi = vqmovun_s32(weighted_s0_hi);
+            uint16x4_t t1_lo = vqmovun_s32(weighted_s1_lo);
+            uint16x4_t t1_hi = vqmovun_s32(weighted_s1_hi);
+            uint16x8_t d0 = vminq_u16(vcombine_u16(t0_lo, t0_hi), vdupq_n_u16(PIXEL_MAX));
+            uint16x8_t d1 = vminq_u16(vcombine_u16(t1_lo, t1_hi), vdupq_n_u16(PIXEL_MAX));
+
+            vst1q_u16(dst + w + 0, d0);
+            vst1q_u16(dst + w + 8, d1);
+            w += 16;
         }
-        for (; x < bx; x += 2)
+        while (w != width);
+
+        src += stride;
+        dst += stride;
+    }
+    while (--height != 0);
+
+#else
+    // Re-arrange the shift operations.
+    // Then, hoist the right shift out of the loop if BSF(w0) >= shift - correction.
+    // Orig: (((srcx << correction) * w0 + round) >> shift) + offset.
+    // New: (srcx * (w0 >> shift - correction)) + (round >> shift) + offset.
+    // (round >> shift) is always zero since round = 1 << (shift - 1).
+
+    unsigned long id;
+    BSF(id, w0);
+
+    if ((int)id >= shift - correction)
+    {
+        w0 >>= shift - correction;
+
+        do
         {
-            dstx + 0 = x265_clip((src0x + 0 + src1x + 0 + offset) >> shiftNum);
-            dstx + 1 = x265_clip((src0x + 1 + src1x + 1 + offset) >> shiftNum);
+            int w = 0;
+            do
+            {
+                uint8x16_t s = vld1q_u8(src + w);
+                int16x8_t weighted_s0 = vreinterpretq_s16_u16(
+                    vmlal_u8(vdupq_n_u16(offset), vget_low_u8(s), vdup_n_u8(w0)));
+                int16x8_t weighted_s1 = vreinterpretq_s16_u16(
+                    vmlal_u8(vdupq_n_u16(offset), vget_high_u8(s), vdup_n_u8(w0)));
+                uint8x8_t d0 = vqmovun_s16(weighted_s0);
+                uint8x8_t d1 = vqmovun_s16(weighted_s1);
+
+                vst1q_u8(dst + w, vcombine_u8(d0, d1));
+                w += 16;
+            }
+            while (w != width);
+
+            src += stride;
+            dst += stride;
         }
+        while (--height != 0);
+    }
+    else // Keep rounding shifts within the loop.
+    {
+        int16x8_t corrected_shift = vdupq_n_s16(correction - shift);
 
-        src0 += src0Stride;
-        src1 += src1Stride;
-        dst  += dstStride;
+        do
+        {
+            int w = 0;
+            do
+            {
+                uint8x16_t s = vld1q_u8(src + w);
+                int16x8_t weighted_s0 =
+                    vreinterpretq_s16_u16(vmull_u8(vget_low_u8(s), vdup_n_u8(w0)));
+                int16x8_t weighted_s1 =
+                    vreinterpretq_s16_u16(vmull_u8(vget_high_u8(s), vdup_n_u8(w0)));
+                weighted_s0 = vrshlq_s16(weighted_s0, corrected_shift);
+                weighted_s1 = vrshlq_s16(weighted_s1, corrected_shift);
+                weighted_s0 = vaddq_s16(weighted_s0, vdupq_n_s16(offset));
+                weighted_s1 = vaddq_s16(weighted_s1, vdupq_n_s16(offset));
+                uint8x8_t d0 = vqmovun_s16(weighted_s0);
+                uint8x8_t d1 = vqmovun_s16(weighted_s1);
+
+                vst1q_u8(dst + w, vcombine_u8(d0, d1));
+                w += 16;
+            }
+            while (w != width);
+
+            src += stride;
+            dst += stride;
+        }
+        while (--height != 0);
     }
+#endif
 }
 
 template<int lx, int ly>
@@ -1431,56 +1551,250 @@
     X265_CHECK(((intptr_t)src & 15) == 0, "src alignment error\n");
     X265_CHECK(shift >= 0, "invalid shift\n");
 
-    for (int i = 0; i < size; i++)
+    for (int h = 0; h < size; h++)
     {
-        int j = 0;
-        for (; (j + 8) <= size; j += 8)
+        for (int w = 0; w + 16 <= size; w += 16)
+        {
+            int16x8_t s0_lo = vld1q_s16(src + w);
+            int16x8_t s0_hi = vld1q_s16(src + w + 8);
+            int16x8_t d0_lo = vshlq_s16(s0_lo, vdupq_n_s16(shift));
+            int16x8_t d0_hi = vshlq_s16(s0_hi, vdupq_n_s16(shift));
+            vst1q_s16(dst + w, d0_lo);
+            vst1q_s16(dst + w + 8, d0_hi);
+        }
+        if (size == 8)
         {
-            vst1q_s16(dst + j, vshlq_s16(vld1q_s16(src + j), vdupq_n_s16(shift)));
+            int16x8_t s0 = vld1q_s16(src);
+            int16x8_t d0 = vshlq_s16(s0, vdupq_n_s16(shift));
+            vst1q_s16(dst, d0);
         }
-        for (; j < size; j++)
+        if (size == 4)
         {
-            dstj = srcj << shift;
+            int16x4_t s0 = vld1_s16(src);
+            int16x4_t d0 = vshl_s16(s0, vdup_n_s16(shift));
+            vst1_s16(dst, d0);
         }
+
         src += size;
         dst += dstStride;
     }
 }
 
+template<int size>
+void cpy1Dto2D_shr_neon(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
+{
+    X265_CHECK((((intptr_t)dst | (dstStride * sizeof(*dst))) & 15) == 0 || size == 4, "dst alignment error\n");
+    X265_CHECK(((intptr_t)src & 15) == 0, "src alignment error\n");
+    X265_CHECK(shift > 0, "invalid shift\n");
 
+    for (int h = 0; h < size; h++)
+    {
+        for (int w = 0; w + 16 <= size; w += 16)
+        {
+            int16x8_t s0_lo = vld1q_s16(src + w);
+            int16x8_t s0_hi = vld1q_s16(src + w + 8);
+            int16x8_t d0_lo = vrshlq_s16(s0_lo, vdupq_n_s16(-shift));
+            int16x8_t d0_hi = vrshlq_s16(s0_hi, vdupq_n_s16(-shift));
+            vst1q_s16(dst + w, d0_lo);
+            vst1q_s16(dst + w + 8, d0_hi);
+        }
+        if (size == 8)
+        {
+            int16x8_t s0 = vld1q_s16(src);
+            int16x8_t d0 = vrshlq_s16(s0, vdupq_n_s16(-shift));
+            vst1q_s16(dst, d0);
+        }
+        if (size == 4)
+        {
+            int16x4_t s0 = vld1_s16(src);
+            int16x4_t d0 = vrshl_s16(s0, vdup_n_s16(-shift));
+            vst1_s16(dst, d0);
+        }
+
+        src += size;
+        dst += dstStride;
+    }
+}
+
+#if HIGH_BIT_DEPTH
 template<int size>
-uint64_t pixel_var_neon(const uint8_t *pix, intptr_t i_stride)
+uint64_t pixel_var_neon(const uint16_t *pix, intptr_t i_stride)
 {
-    uint32_t sum = 0, sqr = 0;
+    // w * h * (2^BIT_DEPTH) <= (2^ACC_DEPTH) * (no. of ACC) * ACC_WIDTH
+    // w * h * (2^BIT_DEPTH) * (2^BIT_DEPTH) <= (2^ACC_DEPTH) * (no. of ACC) * ACC_WIDTH
+    // Minimum requirements to avoid overflow:
+    // 1 uint32x4_t sum acc, 2 uint32x4_t sqr acc for 12-bit 64x64.
+    // 1 uint32x4_t sum acc, 1 uint32x4_t sqr acc for 10/12-bit 32x32 and 10-bit 64x64.
+    // 2 uint16x8_t sum acc, 1 uint32x4_t sqr acc for 10/12-bit 16x16 block sizes.
+    // 1 uint16x8_t sum acc, 1 uint32x4_t sqr acc for 10/12-bit 4x4, 8x8 block sizes.
+    if (size > 16)
+    {
+        uint32x4_t sum2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+        uint32x4_t sqr2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+        for (int h = 0; h < size; ++h)
+        {
+            for (int w = 0; w + 16 <= size; w += 16)
+            {
+                uint16x8_t s2;
+                load_u16x8xn<2>(pix + w, 8, s);
+
+                sum0 = vpadalq_u16(sum0, s0);
+                sum1 = vpadalq_u16(sum1, s1);
 
-    uint32x4_t vsqr = vdupq_n_u32(0);
+                sqr0 = vmlal_u16(sqr0, vget_low_u16(s0), vget_low_u16(s0));
+                sqr0 = vmlal_u16(sqr0, vget_high_u16(s0), vget_high_u16(s0));
+                sqr1 = vmlal_u16(sqr1, vget_low_u16(s1), vget_low_u16(s1));
+                sqr1 = vmlal_u16(sqr1, vget_high_u16(s1), vget_high_u16(s1));
+            }
+
+            pix += i_stride;
+        }
 
-    for (int y = 0; y < size; y++)
+        sum0 = vaddq_u32(sum0, sum1);
+        sqr0 = vaddq_u32(sqr0, sqr1);
+
+        return vaddvq_u32(sum0) + (vaddlvq_u32(sqr0) << 32);
+    }
+    if (size == 16)
     {
-        int x = 0;
-        uint16x8_t vsum = vdupq_n_u16(0);
-        for (; (x + 8) <= size; x += 8)
+        uint16x8_t sum2 = { vdupq_n_u16(0), vdupq_n_u16(0) };
+        uint32x4_t sqr2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+        for (int h = 0; h < size; ++h)
         {
-            uint16x8_t in;
-            in = vmovl_u8(vld1_u8(pix + x));
-            vsum = vaddq_u16(vsum, in);
-            vsqr = vmlal_u16(vsqr, vget_low_u16(in), vget_low_u16(in));
-            vsqr = vmlal_high_u16(vsqr, in, in);
+            uint16x8_t s2;
+            load_u16x8xn<2>(pix, 8, s);
+
+            sum0 = vaddq_u16(sum0, s0);
+            sum1 = vaddq_u16(sum1, s1);
+
+            sqr0 = vmlal_u16(sqr0, vget_low_u16(s0), vget_low_u16(s0));
+            sqr0 = vmlal_u16(sqr0, vget_high_u16(s0), vget_high_u16(s0));
+            sqr1 = vmlal_u16(sqr1, vget_low_u16(s1), vget_low_u16(s1));
+            sqr1 = vmlal_u16(sqr1, vget_high_u16(s1), vget_high_u16(s1));
+
+            pix += i_stride;
         }
-        for (; x < size; x++)
+
+        uint32x4_t sum_u32 = vpaddlq_u16(sum0);
+        sum_u32 = vpadalq_u16(sum_u32, sum1);
+        sqr0 = vaddq_u32(sqr0, sqr1);
+
+        return vaddvq_u32(sum_u32) + (vaddlvq_u32(sqr0) << 32);
+    }
+    if (size == 8)
+    {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint32x4_t sqr = vdupq_n_u32(0);
+
+        for (int h = 0; h < size; ++h)
         {
-            sum += pixx;
-            sqr += pixx * pixx;
+            uint16x8_t s = vld1q_u16(pix);
+
+            sum = vaddq_u16(sum, s);
+            sqr = vmlal_u16(sqr, vget_low_u16(s), vget_low_u16(s));
+            sqr = vmlal_u16(sqr, vget_high_u16(s), vget_high_u16(s));
+
+            pix += i_stride;
         }
 
-        sum += vaddvq_u16(vsum);
+        return vaddlvq_u16(sum) + (vaddlvq_u32(sqr) << 32);
+    }
+    if (size == 4) {
+        uint16x4_t sum = vdup_n_u16(0);
+        uint32x4_t sqr = vdupq_n_u32(0);
 
-        pix += i_stride;
+        for (int h = 0; h < size; ++h)
+        {
+            uint16x4_t s = vld1_u16(pix);
+
+            sum = vadd_u16(sum, s);
+            sqr = vmlal_u16(sqr, s, s);
+
+            pix += i_stride;
+        }
+
+        return vaddv_u16(sum) + (vaddlvq_u32(sqr) << 32);
     }
-    sqr += vaddvq_u32(vsqr);
-    return sum + ((uint64_t)sqr << 32);
 }
 
+#else // !HIGH_BIT_DEPTH
+template<int size>
+uint64_t pixel_var_neon(const uint8_t *pix, intptr_t i_stride)
+{
+    if (size >= 16)
+    {
+        uint16x8_t sum2 = { vdupq_n_u16(0), vdupq_n_u16(0) };
+        uint32x4_t sqr2 = { vdupq_n_u32(0), vdupq_n_u32(0) };
+
+        for (int h = 0; h < size; h += 2)
+        {
+            for (int w = 0; w + 16 <= size; w += 16)
+            {
+                uint8x16_t s2;
+                load_u8x16xn<2>(pix + w, i_stride, s);
+
+                sum0 = vpadalq_u8(sum0, s0);
+                sum1 = vpadalq_u8(sum1, s1);
+
+                uint16x8_t sqr_lo = vmull_u8(vget_low_u8(s0), vget_low_u8(s0));
+                uint16x8_t sqr_hi = vmull_u8(vget_high_u8(s0), vget_high_u8(s0));
+                sqr0 = vpadalq_u16(sqr0, sqr_lo);
+                sqr0 = vpadalq_u16(sqr0, sqr_hi);
+
+                sqr_lo = vmull_u8(vget_low_u8(s1), vget_low_u8(s1));
+                sqr_hi = vmull_u8(vget_high_u8(s1), vget_high_u8(s1));
+                sqr1 = vpadalq_u16(sqr1, sqr_lo);
+                sqr1 = vpadalq_u16(sqr1, sqr_hi);
+            }
+
+            pix += 2 * i_stride;
+        }
+
+        uint32x4_t sum_u32 = vpaddlq_u16(sum0);
+        sum_u32 = vpadalq_u16(sum_u32, sum1);
+        sqr0 = vaddq_u32(sqr0, sqr1);
+
+        return vaddvq_u32(sum_u32) + (vaddlvq_u32(sqr0) << 32);
+    }
+    if (size == 8)
+    {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint32x4_t sqr = vdupq_n_u32(0);
+
+        for (int h = 0; h < size; ++h)
+        {
+            uint8x8_t s = vld1_u8(pix);
+
+            sum = vaddw_u8(sum, s);
+            sqr = vpadalq_u16(sqr, vmull_u8(s, s));
+
+            pix += i_stride;
+        }
+
+        return vaddvq_u16(sum) + (vaddlvq_u32(sqr) << 32);
+    }
+    if (size == 4)
+    {
+        uint16x8_t sum = vdupq_n_u16(0);
+        uint32x4_t sqr = vdupq_n_u32(0);
+
+        for (int h = 0; h < size; h += 2)
+        {
+            uint8x8_t s = load_u8x4x2(pix, i_stride);
+
+            sum = vaddw_u8(sum, s);
+            sqr = vpadalq_u16(sqr, vmull_u8(s, s));
+
+            pix += 2 * i_stride;
+        }
+
+        return vaddvq_u16(sum) + (vaddlvq_u32(sqr) << 32);
+    }
+}
+#endif // HIGH_BIT_DEPTH
+
 template<int blockSize>
 void getResidual_neon(const pixel *fenc, const pixel *pred, int16_t *residual, intptr_t stride)
 {
@@ -1497,24 +1811,144 @@
             vfenc = vmovl_u8(vld1_u8(fenc + x));
             vpred = vmovl_u8(vld1_u8(pred + x));
 #endif
-            int16x8_t res = vreinterpretq_s16_u16(vsubq_u16(vfenc, vpred));
-            vst1q_s16(residual + x, res);
-        }
-        for (; x < blockSize; x++)
-        {
-            residualx = static_cast<int16_t>(fencx) - static_cast<int16_t>(predx);
-        }
-        fenc += stride;
-        residual += stride;
-        pred += stride;
-    }
+            int16x8_t res = vreinterpretq_s16_u16(vsubq_u16(vfenc, vpred));
+            vst1q_s16(residual + x, res);
+        }
+        for (; x < blockSize; x++)
+        {
+            residualx = static_cast<int16_t>(fencx) - static_cast<int16_t>(predx);
+        }
+        fenc += stride;
+        residual += stride;
+        pred += stride;
+    }
+}
+
+#if HIGH_BIT_DEPTH
+static inline int calc_energy_8x8(const uint16_t *source, intptr_t sstride)
+{
+    uint16x8_t s8;
+    load_u16x8xn<8>(source, sstride, s);
+
+    int16x8_t in8, temp8;
+
+    in0 = vreinterpretq_s16_u16(vaddq_u16(s0, s1));
+    in1 = vreinterpretq_s16_u16(vaddq_u16(s2, s3));
+    in2 = vreinterpretq_s16_u16(vaddq_u16(s4, s5));
+    in3 = vreinterpretq_s16_u16(vaddq_u16(s6, s7));
+    in4 = vreinterpretq_s16_u16(vsubq_u16(s0, s1));
+    in5 = vreinterpretq_s16_u16(vsubq_u16(s2, s3));
+    in6 = vreinterpretq_s16_u16(vsubq_u16(s4, s5));
+    in7 = vreinterpretq_s16_u16(vsubq_u16(s6, s7));
+
+    hadamard_4_v(in, temp);
+    hadamard_4_v(in + 4, temp + 4);
+
+    // The first line after the vertical hadamard transform contains the sum of coefficients.
+    int sum = vaddlvq_s16(temp0) >> 2;
+
+#if X265_DEPTH == 10
+    uint16x8_t sa8_out4;
+
+    hadamard_8_h(temp, sa8_out);
+
+    uint32x4_t res = vpaddlq_u16(sa8_out0);
+    res = vpadalq_u16(res, sa8_out1);
+    res = vpadalq_u16(res, sa8_out2);
+    res = vpadalq_u16(res, sa8_out3);
+#else // X265_DEPTH == 12
+    uint32x4_t sa8_out4;
+
+    hadamard_8_h(temp, sa8_out);
+
+    sa8_out0 = vaddq_u32(sa8_out0, sa8_out1);
+    sa8_out2 = vaddq_u32(sa8_out2, sa8_out3);
+    uint32x4_t res = vaddq_u32(sa8_out0, sa8_out2);
+#endif // X265_DEPTH == 10
+
+    int sa8 = (vaddvq_u32(res) + 1) >> 1;
+
+    return sa8 - sum;
+}
+
+#else // !HIGH_BIT_DEPTH
+static inline int calc_energy_8x8(const uint8_t *source, intptr_t sstride)
+{
+    uint8x8_t s8;
+    load_u8x8xn<8>(source, sstride, s);
+
+    int16x8_t in8, temp8;
+
+    in0 = vreinterpretq_s16_u16(vaddl_u8(s0, s1));
+    in1 = vreinterpretq_s16_u16(vaddl_u8(s2, s3));
+    in2 = vreinterpretq_s16_u16(vaddl_u8(s4, s5));
+    in3 = vreinterpretq_s16_u16(vaddl_u8(s6, s7));
+    in4 = vreinterpretq_s16_u16(vsubl_u8(s0, s1));
+    in5 = vreinterpretq_s16_u16(vsubl_u8(s2, s3));
+    in6 = vreinterpretq_s16_u16(vsubl_u8(s4, s5));
+    in7 = vreinterpretq_s16_u16(vsubl_u8(s6, s7));
+
+    hadamard_4_v(in, temp);
+    hadamard_4_v(in + 4, temp + 4);
+
+    // The first line after the vertical hadamard transform contains the sum of coefficients.
+    int sum = vaddvq_s16(temp0) >> 2;
+
+    uint16x8_t sa8_out4;
+    hadamard_8_h(temp, sa8_out);
+
+    uint16x8_t res = vaddq_u16(sa8_out0, sa8_out1);
+    res = vaddq_u16(res, sa8_out2);
+    res = vaddq_u16(res, sa8_out3);
+
+    int sa8 = (vaddlvq_u16(res) + 1) >> 1;
+
+    return sa8 - sum;
+}
+
+#endif // HIGH_BIT_DEPTH
+
+static inline int calc_energy_4x4(const pixel *source, intptr_t sstride)
+{
+#if HIGH_BIT_DEPTH
+    uint16x4_t s4;
+    load_u16x4xn<4>(source, sstride, s);
+
+    uint16x8_t s01 = vcombine_u16(s0, s1);
+    uint16x8_t s23 = vcombine_u16(s2, s3);
+
+    int16x8_t s01_23 = vreinterpretq_s16_u16(vaddq_u16(s01, s23));
+    int16x8_t d01_23 = vreinterpretq_s16_u16(vsubq_u16(s01, s23));
+#else
+    uint8x8_t s2;
+    s0 = load_u8x4x2(source + 0 * sstride, sstride);
+    s1 = load_u8x4x2(source + 2 * sstride, sstride);
+
+    int16x8_t s01_23 = vreinterpretq_s16_u16(vaddl_u8(s0, s1));
+    int16x8_t d01_23 = vreinterpretq_s16_u16(vsubl_u8(s0, s1));
+#endif
+
+    // The first line after the vertical hadamard transform contains the sum of coefficients.
+    int sum = vaddvq_u16(vreinterpretq_u16_s16(s01_23)) >> 2;
+
+    int16x8_t t0, t1;
+
+    transpose_s16_s64x2(&t0, &t1, s01_23, d01_23);
+    sumsubq_s16(&s01_23, &d01_23, t0, t1);
+
+    transpose_s16_s16x2(&t0, &t1, s01_23, d01_23);
+    sumsubq_s16(&s01_23, &d01_23, t0, t1);
+
+    transpose_s16_s32x2(&t0, &t1, s01_23, d01_23);
+
+    int sat = vaddvq_u16(max_abs_s16(t0, t1));
+
+    return sat - sum;
 }
 
 template<int size>
 int psyCost_pp_neon(const pixel *source, intptr_t sstride, const pixel *recon, intptr_t rstride)
 {
-    static pixel zeroBuf8 /* = { 0 } */;
-
     if (size)
     {
         int dim = 1 << (size + 2);
@@ -1523,11 +1957,8 @@
         {
             for (int j = 0; j < dim; j += 8)
             {
-                /* AC energy, measured by sa8d (AC + DC) minus SAD (DC) */
-                int sourceEnergy = pixel_sa8d_8x8_neon(source + i * sstride + j, sstride, zeroBuf, 0) -
-                                   (sad_pp_neon<8, 8>(source + i * sstride + j, sstride, zeroBuf, 0) >> 2);
-                int reconEnergy =  pixel_sa8d_8x8_neon(recon + i * rstride + j, rstride, zeroBuf, 0) -
-                                   (sad_pp_neon<8, 8>(recon + i * rstride + j, rstride, zeroBuf, 0) >> 2);
+                int sourceEnergy = calc_energy_8x8(source + i * sstride + j, sstride);
+                int reconEnergy = calc_energy_8x8(recon + i * rstride + j, rstride);
 
                 totEnergy += abs(sourceEnergy - reconEnergy);
             }
@@ -1536,11 +1967,9 @@
     }
     else
     {
-        /* 4x4 is too small for sa8d */
-        int sourceEnergy = pixel_satd_4x4_neon(source, sstride, zeroBuf, 0) - (sad_pp_neon<4, 4>(source, sstride, zeroBuf,
-                           0) >> 2);
-        int reconEnergy = pixel_satd_4x4_neon(recon, rstride, zeroBuf, 0) - (sad_pp_neon<4, 4>(recon, rstride, zeroBuf,
-                          0) >> 2);
+        int sourceEnergy = calc_energy_4x4(source, sstride);
+        int reconEnergy = calc_energy_4x4(recon, rstride);
+
         return abs(sourceEnergy - reconEnergy);
     }
 }
@@ -1548,7 +1977,7 @@
 
 template<int w, int h>
 // Calculate sa8d in blocks of 8x8
-int sa8d8(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
+int sa8d8_neon(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
 {
     int cost = 0;
 
@@ -1563,7 +1992,7 @@
 
 template<int w, int h>
 // Calculate sa8d in blocks of 16x16
-int sa8d16(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
+int sa8d16_neon(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
 {
     int cost = 0;
 
@@ -1583,11 +2012,29 @@
     X265_CHECK((((intptr_t)src | (srcStride * sizeof(*src))) & 15) == 0 || size == 4, "src alignment error\n");
     X265_CHECK(shift >= 0, "invalid shift\n");
 
-    for (int i = 0; i < size; i++)
+    for (int h = 0; h < size; h++)
     {
-        for (int j = 0; j < size; j++)
+        int w = 0;
+        for (; w + 16 <= size; w += 16)
+        {
+            int16x8_t a0_lo = vld1q_s16(src + w);
+            int16x8_t a0_hi = vld1q_s16(src + w + 8);
+            int16x8_t d0_lo = vshlq_s16(a0_lo, vdupq_n_s16(shift));
+            int16x8_t d0_hi = vshlq_s16(a0_hi, vdupq_n_s16(shift));
+            vst1q_s16(dst + w, d0_lo);
+            vst1q_s16(dst + w + 8, d0_hi);
+        }
+        if (size == 8)
         {
-            dstj = srcj << shift;
+            int16x8_t a0 = vld1q_s16(src + w);
+            int16x8_t d0 = vshlq_s16(a0, vdupq_n_s16(shift));
+            vst1q_s16(dst + w, d0);
+        }
+        if (size == 4)
+        {
+            int16x4_t a0 = vld1_s16(src + w);
+            int16x4_t d0 = vshl_s16(a0, vdup_n_s16(shift));
+            vst1_s16(dst + w, d0);
         }
 
         src += srcStride;
@@ -1595,44 +2042,100 @@
     }
 }
 
+template<int size>
+void cpy2Dto1D_shr_neon(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
+{
+    X265_CHECK(((intptr_t)dst & 15) == 0, "dst alignment error\n");
+    X265_CHECK((((intptr_t)src | (srcStride * sizeof(*src))) & 15) == 0 || size == 4, "src alignment error\n");
+    X265_CHECK(shift > 0, "invalid shift\n");
+
+    for (int h = 0; h < size; h++)
+    {
+        for (int w = 0; w + 16 <= size; w += 16)
+        {
+            int16x8_t s0_lo = vld1q_s16(src + w);
+            int16x8_t s0_hi = vld1q_s16(src + w + 8);
+            int16x8_t d0_lo = vrshlq_s16(s0_lo, vdupq_n_s16(-shift));
+            int16x8_t d0_hi = vrshlq_s16(s0_hi, vdupq_n_s16(-shift));
+            vst1q_s16(dst + w, d0_lo);
+            vst1q_s16(dst + w + 8, d0_hi);
+        }
+        if (size == 8)
+        {
+            int16x8_t s0 = vld1q_s16(src);
+            int16x8_t d0 = vrshlq_s16(s0, vdupq_n_s16(-shift));
+            vst1q_s16(dst, d0);
+        }
+        if (size == 4)
+        {
+            int16x4_t s0 = vld1_s16(src);
+            int16x4_t d0 = vrshl_s16(s0, vdup_n_s16(-shift));
+            vst1_s16(dst, d0);
+        }
+
+        src += srcStride;
+        dst += size;
+    }
+}
 
 template<int w, int h>
-// calculate satd in blocks of 4x4
 int satd4_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
 {
     int satd = 0;
 
-    for (int row = 0; row < h; row += 4)
-        for (int col = 0; col < w; col += 4)
-            satd += pixel_satd_4x4_neon(pix1 + row * stride_pix1 + col, stride_pix1,
-                                        pix2 + row * stride_pix2 + col, stride_pix2);
+    if (w == 4 && h == 4) {
+        satd = pixel_satd_4x4_neon(pix1, stride_pix1, pix2, stride_pix2);
+    } else {
+        for (int row = 0; row < h; row += 8)
+            for (int col = 0; col < w; col += 4)
+                satd += pixel_satd_4x8_neon(pix1 + row * stride_pix1 + col, stride_pix1,
+                                            pix2 + row * stride_pix2 + col, stride_pix2);
+    }
 
     return satd;
 }
 
 template<int w, int h>
-// calculate satd in blocks of 8x4
 int satd8_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
 {
     int satd = 0;
 
-    if (((w | h) & 15) == 0)
+    if (w % 16 == 0 && h % 16 == 0)
     {
         for (int row = 0; row < h; row += 16)
             for (int col = 0; col < w; col += 16)
                 satd += pixel_satd_16x16_neon(pix1 + row * stride_pix1 + col, stride_pix1,
                                               pix2 + row * stride_pix2 + col, stride_pix2);
-
     }
-    else if (((w | h) & 7) == 0)
+    else if (w % 8 == 0 && h % 16 == 0)
+    {
+        for (int row = 0; row < h; row += 16)
+            for (int col = 0; col < w; col += 8)
+                satd += pixel_satd_8x16_neon(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 16 == 0 && h % 8 == 0)
+    {
+        for (int row = 0; row < h; row += 8)
+            for (int col = 0; col < w; col += 16)
+                satd += pixel_satd_16x8_neon(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 16 == 0 && h % 4 == 0)
+    {
+        for (int row = 0; row < h; row += 4)
+            for (int col = 0; col < w; col += 16)
+                satd += pixel_satd_16x4_neon(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 8 == 0 && h % 8 == 0)
     {
         for (int row = 0; row < h; row += 8)
             for (int col = 0; col < w; col += 8)
                 satd += pixel_satd_8x8_neon(pix1 + row * stride_pix1 + col, stride_pix1,
                                             pix2 + row * stride_pix2 + col, stride_pix2);
-
     }
-    else
+    else // w multiple of 8, h multiple of 4
     {
         for (int row = 0; row < h; row += 4)
             for (int col = 0; col < w; col += 8)
@@ -1684,34 +2187,6 @@
 }
 
 
-template<int size>
-sse_t pixel_ssd_s_neon(const int16_t *a, intptr_t dstride)
-{
-    sse_t sum = 0;
-
-
-    int32x4_t vsum = vdupq_n_s32(0);
-
-    for (int y = 0; y < size; y++)
-    {
-        int x = 0;
-
-        for (; (x + 8) <= size; x += 8)
-        {
-            int16x8_t in = vld1q_s16(a + x);
-            vsum = vmlal_s16(vsum, vget_low_s16(in), vget_low_s16(in));
-            vsum = vmlal_high_s16(vsum, (in), (in));
-        }
-        for (; x < size; x++)
-        {
-            sum += ax * ax;
-        }
-
-        a += dstride;
-    }
-    return sum + vaddvq_s32(vsum);
-}
-
 
 };
 
@@ -1728,9 +2203,6 @@
     p.puLUMA_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.addAvgNONALIGNED = addAvg_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.addAvgALIGNED = addAvg_neon<W, H>; \
-    p.puLUMA_ ## W ## x ## H.sad = sad_pp_neon<W, H>; \
-    p.puLUMA_ ## W ## x ## H.sad_x3 = sad_x3_neon<W, H>; \
-    p.puLUMA_ ## W ## x ## H.sad_x4 = sad_x4_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.pixelavg_ppNONALIGNED = pixelavg_pp_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.pixelavg_ppALIGNED = pixelavg_pp_neon<W, H>;
 
@@ -1744,25 +2216,45 @@
     p.puLUMA_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.addAvgNONALIGNED = addAvg_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.addAvgALIGNED = addAvg_neon<W, H>; \
-    p.puLUMA_ ## W ## x ## H.sad_x3 = sad_x3_neon<W, H>; \
-    p.puLUMA_ ## W ## x ## H.sad_x4 = sad_x4_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.pixelavg_ppNONALIGNED = pixelavg_pp_neon<W, H>; \
     p.puLUMA_ ## W ## x ## H.pixelavg_ppALIGNED = pixelavg_pp_neon<W, H>;
 #endif // !(HIGH_BIT_DEPTH)
 
+#if HIGH_BIT_DEPTH
 #define LUMA_CU(W, H) \
     p.cuBLOCK_ ## W ## x ## H.sub_ps        = pixel_sub_ps_neon<W, H>; \
     p.cuBLOCK_ ## W ## x ## H.add_psNONALIGNED    = pixel_add_ps_neon<W, H>; \
     p.cuBLOCK_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.blockfill_sNONALIGNED = blockfill_s_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.blockfill_sALIGNED    = blockfill_s_neon<W>; \
     p.cuBLOCK_ ## W ## x ## H.copy_pp       = blockcopy_pp_neon<W, H>; \
-    p.cuBLOCK_ ## W ## x ## H.copy_ps       = blockcopy_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.copy_ss       = blockcopy_ss_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.cpy2Dto1D_shl = cpy2Dto1D_shl_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.cpy1Dto2D_shlNONALIGNED = cpy1Dto2D_shl_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.cpy1Dto2D_shlALIGNED = cpy1Dto2D_shl_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.psy_cost_pp   = psyCost_pp_neon<BLOCK_ ## W ## x ## H>; \
+    p.cuBLOCK_ ## W ## x ## H.transpose     = transpose_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.var           = pixel_var_neon<W>;
+#else  // !HIGH_BIT_DEPTH
+#define LUMA_CU(W, H) \
+    p.cuBLOCK_ ## W ## x ## H.sub_ps        = pixel_sub_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.add_psNONALIGNED    = pixel_add_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.blockfill_sNONALIGNED = blockfill_s_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.blockfill_sALIGNED    = blockfill_s_neon<W>; \
     p.cuBLOCK_ ## W ## x ## H.copy_pp       = blockcopy_pp_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.copy_ps       = blockcopy_ps_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.copy_ss       = blockcopy_ss_neon<W, H>; \
+    p.cuBLOCK_ ## W ## x ## H.copy_sp       = blockcopy_sp_neon<W, H>; \
     p.cuBLOCK_ ## W ## x ## H.cpy2Dto1D_shl = cpy2Dto1D_shl_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.cpy2Dto1D_shr = cpy2Dto1D_shr_neon<W>; \
     p.cuBLOCK_ ## W ## x ## H.cpy1Dto2D_shlNONALIGNED = cpy1Dto2D_shl_neon<W>; \
     p.cuBLOCK_ ## W ## x ## H.cpy1Dto2D_shlALIGNED = cpy1Dto2D_shl_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.cpy1Dto2D_shr = cpy1Dto2D_shr_neon<W>; \
     p.cuBLOCK_ ## W ## x ## H.psy_cost_pp   = psyCost_pp_neon<BLOCK_ ## W ## x ## H>; \
-    p.cuBLOCK_ ## W ## x ## H.transpose     = transpose_neon<W>;
-
+    p.cuBLOCK_ ## W ## x ## H.transpose     = transpose_neon<W>; \
+    p.cuBLOCK_ ## W ## x ## H.var           = pixel_var_neon<W>;
+#endif // HIGH_BIT_DEPTH
 
     LUMA_PU_S(4, 4);
     LUMA_PU_S(8, 8);
@@ -1789,58 +2281,32 @@
     LUMA_PU(48, 64);
     LUMA_PU(64, 16);
     LUMA_PU(16, 64);
-    
-#if defined(__APPLE__)
-    p.puLUMA_4x4.sad = sad_pp_neon<4, 4>;
-    p.puLUMA_4x8.sad = sad_pp_neon<4, 8>;
-    p.puLUMA_4x16.sad = sad_pp_neon<4, 16>;
-#endif // defined(__APPLE__)
-    p.puLUMA_8x4.sad = sad_pp_neon<8, 4>;
-    p.puLUMA_8x8.sad = sad_pp_neon<8, 8>;
-    p.puLUMA_8x16.sad = sad_pp_neon<8, 16>;
-    p.puLUMA_8x32.sad = sad_pp_neon<8, 32>;
-
-#if !(HIGH_BIT_DEPTH)
-    p.puLUMA_4x4.sad_x3 = sad_x3_neon<4, 4>;
-    p.puLUMA_4x4.sad_x4 = sad_x4_neon<4, 4>;
-    p.puLUMA_4x8.sad_x3 = sad_x3_neon<4, 8>;
-    p.puLUMA_4x8.sad_x4 = sad_x4_neon<4, 8>;
-    p.puLUMA_4x16.sad_x3 = sad_x3_neon<4, 16>;
-    p.puLUMA_4x16.sad_x4 = sad_x4_neon<4, 16>;
-#endif // !(HIGH_BIT_DEPTH)
 
-    p.puLUMA_4x4.satd   = pixel_satd_4x4_neon;
-    p.puLUMA_8x4.satd   = pixel_satd_8x4_neon;
-    
+    p.puLUMA_4x4.satd   = satd4_neon<4, 4>;
+    p.puLUMA_4x8.satd   = satd4_neon<4, 8>;
+    p.puLUMA_4x16.satd  = satd4_neon<4, 16>;
+    p.puLUMA_8x4.satd   = satd8_neon<8, 4>;
     p.puLUMA_8x8.satd   = satd8_neon<8, 8>;
-    p.puLUMA_16x16.satd = satd8_neon<16, 16>;
-    p.puLUMA_16x8.satd  = satd8_neon<16, 8>;
     p.puLUMA_8x16.satd  = satd8_neon<8, 16>;
-    p.puLUMA_16x12.satd = satd8_neon<16, 12>;
+    p.puLUMA_8x32.satd  = satd8_neon<8, 32>;
+    p.puLUMA_12x16.satd = satd4_neon<12, 16>;
     p.puLUMA_16x4.satd  = satd8_neon<16, 4>;
-    p.puLUMA_32x32.satd = satd8_neon<32, 32>;
-    p.puLUMA_32x16.satd = satd8_neon<32, 16>;
+    p.puLUMA_16x8.satd  = satd8_neon<16, 8>;
+    p.puLUMA_16x12.satd = satd8_neon<16, 12>;
+    p.puLUMA_16x16.satd = satd8_neon<16, 16>;
     p.puLUMA_16x32.satd = satd8_neon<16, 32>;
-    p.puLUMA_32x24.satd = satd8_neon<32, 24>;
+    p.puLUMA_16x64.satd = satd8_neon<16, 64>;
     p.puLUMA_24x32.satd = satd8_neon<24, 32>;
     p.puLUMA_32x8.satd  = satd8_neon<32, 8>;
-    p.puLUMA_8x32.satd  = satd8_neon<8, 32>;
-    p.puLUMA_64x64.satd = satd8_neon<64, 64>;
-    p.puLUMA_64x32.satd = satd8_neon<64, 32>;
+    p.puLUMA_32x16.satd = satd8_neon<32, 16>;
+    p.puLUMA_32x24.satd = satd8_neon<32, 24>;
+    p.puLUMA_32x32.satd = satd8_neon<32, 32>;
     p.puLUMA_32x64.satd = satd8_neon<32, 64>;
-    p.puLUMA_64x48.satd = satd8_neon<64, 48>;
     p.puLUMA_48x64.satd = satd8_neon<48, 64>;
     p.puLUMA_64x16.satd = satd8_neon<64, 16>;
-    p.puLUMA_16x64.satd = satd8_neon<16, 64>;
-
-#if HIGH_BIT_DEPTH
-    p.puLUMA_4x8.satd   = satd4_neon<4, 8>;
-    p.puLUMA_4x16.satd  = satd4_neon<4, 16>;
-#endif // HIGH_BIT_DEPTH
-
-#if !defined(__APPLE__) || HIGH_BIT_DEPTH
-    p.puLUMA_12x16.satd = satd4_neon<12, 16>;
-#endif // !defined(__APPLE__)
+    p.puLUMA_64x32.satd = satd8_neon<64, 32>;
+    p.puLUMA_64x48.satd = satd8_neon<64, 48>;
+    p.puLUMA_64x64.satd = satd8_neon<64, 64>;
 
 
     LUMA_CU(4, 4);
@@ -1848,22 +2314,6 @@
     LUMA_CU(16, 16);
     LUMA_CU(32, 32);
     LUMA_CU(64, 64);
-    
-#if !(HIGH_BIT_DEPTH)
-    p.cuBLOCK_8x8.var   = pixel_var_neon<8>;
-    p.cuBLOCK_16x16.var = pixel_var_neon<16>;
-#if defined(__APPLE__)
-    p.cuBLOCK_32x32.var   = pixel_var_neon<32>;
-    p.cuBLOCK_64x64.var = pixel_var_neon<64>;
-#endif // defined(__APPLE__)
-#endif // !(HIGH_BIT_DEPTH)
-
-    p.cuBLOCK_16x16.blockfill_sNONALIGNED = blockfill_s_neon<16>; 
-    p.cuBLOCK_16x16.blockfill_sALIGNED    = blockfill_s_neon<16>;
-    p.cuBLOCK_32x32.blockfill_sNONALIGNED = blockfill_s_neon<32>; 
-    p.cuBLOCK_32x32.blockfill_sALIGNED    = blockfill_s_neon<32>;
-    p.cuBLOCK_64x64.blockfill_sNONALIGNED = blockfill_s_neon<64>; 
-    p.cuBLOCK_64x64.blockfill_sALIGNED    = blockfill_s_neon<64>;
 
 
     p.cuBLOCK_4x4.calcresidualNONALIGNED    = getResidual_neon<4>;
@@ -1872,17 +2322,14 @@
     p.cuBLOCK_8x8.calcresidualALIGNED       = getResidual_neon<8>;
     p.cuBLOCK_16x16.calcresidualNONALIGNED  = getResidual_neon<16>;
     p.cuBLOCK_16x16.calcresidualALIGNED     = getResidual_neon<16>;
-    
-#if defined(__APPLE__)
     p.cuBLOCK_32x32.calcresidualNONALIGNED  = getResidual_neon<32>;
     p.cuBLOCK_32x32.calcresidualALIGNED     = getResidual_neon<32>;
-#endif // defined(__APPLE__)
 
-    p.cuBLOCK_4x4.sa8d   = pixel_satd_4x4_neon;
-    p.cuBLOCK_8x8.sa8d   = pixel_sa8d_8x8_neon;
-    p.cuBLOCK_16x16.sa8d = pixel_sa8d_16x16_neon;
-    p.cuBLOCK_32x32.sa8d = sa8d16<32, 32>;
-    p.cuBLOCK_64x64.sa8d = sa8d16<64, 64>;
+    p.cuBLOCK_4x4.sa8d   = satd4_neon<4, 4>;
+    p.cuBLOCK_8x8.sa8d   = sa8d8_neon<8, 8>;
+    p.cuBLOCK_16x16.sa8d = sa8d16_neon<16, 16>;
+    p.cuBLOCK_32x32.sa8d = sa8d16_neon<32, 32>;
+    p.cuBLOCK_64x64.sa8d = sa8d16_neon<64, 64>;
 
 
 #define CHROMA_PU_420(W, H) \
@@ -1891,93 +2338,87 @@
     p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
 
 
-    CHROMA_PU_420(4, 4);
-    CHROMA_PU_420(8, 8);
-    CHROMA_PU_420(16, 16);
-    CHROMA_PU_420(32, 32);
+    CHROMA_PU_420(2, 4);
+    CHROMA_PU_420(2, 8);
     CHROMA_PU_420(4, 2);
-    CHROMA_PU_420(8, 4);
+    CHROMA_PU_420(4, 4);
     CHROMA_PU_420(4, 8);
-    CHROMA_PU_420(8, 6);
     CHROMA_PU_420(6, 8);
+    CHROMA_PU_420(4, 16);
     CHROMA_PU_420(8, 2);
-    CHROMA_PU_420(2, 8);
-    CHROMA_PU_420(16, 8);
-    CHROMA_PU_420(8,  16);
-    CHROMA_PU_420(16, 12);
+    CHROMA_PU_420(8, 4);
+    CHROMA_PU_420(8, 6);
+    CHROMA_PU_420(8, 8);
+    CHROMA_PU_420(8, 16);
+    CHROMA_PU_420(8, 32);
     CHROMA_PU_420(12, 16);
     CHROMA_PU_420(16, 4);
-    CHROMA_PU_420(4,  16);
-    CHROMA_PU_420(32, 16);
+    CHROMA_PU_420(16, 8);
+    CHROMA_PU_420(16, 12);
+    CHROMA_PU_420(16, 16);
     CHROMA_PU_420(16, 32);
-    CHROMA_PU_420(32, 24);
     CHROMA_PU_420(24, 32);
     CHROMA_PU_420(32, 8);
-    CHROMA_PU_420(8,  32);
+    CHROMA_PU_420(32, 16);
+    CHROMA_PU_420(32, 24);
+    CHROMA_PU_420(32, 32);
 
 
 
     p.chromaX265_CSP_I420.puCHROMA_420_2x2.satd   = NULL;
-    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd   = pixel_satd_4x4_neon;
-    p.chromaX265_CSP_I420.puCHROMA_420_8x8.satd   = satd8_neon<8, 8>;
-    p.chromaX265_CSP_I420.puCHROMA_420_16x16.satd = satd8_neon<16, 16>;
-    p.chromaX265_CSP_I420.puCHROMA_420_32x32.satd = satd8_neon<32, 32>;
-
-    p.chromaX265_CSP_I420.puCHROMA_420_4x2.satd   = NULL;
     p.chromaX265_CSP_I420.puCHROMA_420_2x4.satd   = NULL;
-    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd   = pixel_satd_8x4_neon;
-    p.chromaX265_CSP_I420.puCHROMA_420_16x8.satd  = satd8_neon<16, 8>;
-    p.chromaX265_CSP_I420.puCHROMA_420_8x16.satd  = satd8_neon<8, 16>;
-    p.chromaX265_CSP_I420.puCHROMA_420_32x16.satd = satd8_neon<32, 16>;
-    p.chromaX265_CSP_I420.puCHROMA_420_16x32.satd = satd8_neon<16, 32>;
-
-    p.chromaX265_CSP_I420.puCHROMA_420_8x6.satd   = NULL;
-    p.chromaX265_CSP_I420.puCHROMA_420_6x8.satd   = NULL;
-    p.chromaX265_CSP_I420.puCHROMA_420_8x2.satd   = NULL;
     p.chromaX265_CSP_I420.puCHROMA_420_2x8.satd   = NULL;
-    p.chromaX265_CSP_I420.puCHROMA_420_16x12.satd = satd4_neon<16, 12>;
-    p.chromaX265_CSP_I420.puCHROMA_420_16x4.satd  = satd4_neon<16, 4>;
-    p.chromaX265_CSP_I420.puCHROMA_420_32x24.satd = satd8_neon<32, 24>;
-    p.chromaX265_CSP_I420.puCHROMA_420_24x32.satd = satd8_neon<24, 32>;
-    p.chromaX265_CSP_I420.puCHROMA_420_32x8.satd  = satd8_neon<32, 8>;
-    p.chromaX265_CSP_I420.puCHROMA_420_8x32.satd  = satd8_neon<8, 32>;
-    
-#if HIGH_BIT_DEPTH
+    p.chromaX265_CSP_I420.puCHROMA_420_4x2.satd   = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd   = satd4_neon<4, 4>;
     p.chromaX265_CSP_I420.puCHROMA_420_4x8.satd   = satd4_neon<4, 8>;
     p.chromaX265_CSP_I420.puCHROMA_420_4x16.satd  = satd4_neon<4, 16>;
-#endif // HIGH_BIT_DEPTH
-
-#if !defined(__APPLE__) || HIGH_BIT_DEPTH
+    p.chromaX265_CSP_I420.puCHROMA_420_6x8.satd   = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x2.satd   = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd   = satd8_neon<8, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x6.satd   = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x8.satd   = satd8_neon<8, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.satd  = satd8_neon<8, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.satd  = satd8_neon<8, 32>;
     p.chromaX265_CSP_I420.puCHROMA_420_12x16.satd = satd4_neon<12, 16>;
-#endif // !defined(__APPLE__)
-
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.satd  = satd8_neon<16, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.satd  = satd8_neon<16, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.satd = satd8_neon<16, 12>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.satd = satd8_neon<16, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.satd = satd8_neon<16, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.satd = satd8_neon<24, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.satd  = satd8_neon<32, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.satd = satd8_neon<32, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.satd = satd8_neon<32, 24>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.satd = satd8_neon<32, 32>;
 
+#if HIGH_BIT_DEPTH
 #define CHROMA_CU_420(W, H) \
-    p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.sse_pp  = sse_neon<W, H, pixel, pixel>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
-    p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_ps = blockcopy_ps_neon<W, H>; \
+    p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_ss = blockcopy_ss_neon<W, H>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.sub_ps = pixel_sub_ps_neon<W, H>;  \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.add_psNONALIGNED = pixel_add_ps_neon<W, H>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>;
-    
-#define CHROMA_CU_S_420(W, H) \
+#else // !HIGH_BIT_DEPTH
+#define CHROMA_CU_420(W, H) \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_ps = blockcopy_ps_neon<W, H>; \
+    p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_ss = blockcopy_ss_neon<W, H>; \
+    p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.copy_sp = blockcopy_sp_neon<W, H>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.sub_ps = pixel_sub_ps_neon<W, H>;  \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.add_psNONALIGNED = pixel_add_ps_neon<W, H>; \
     p.chromaX265_CSP_I420.cuBLOCK_420_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>;
+#endif // HIGH_BIT_DEPTH
 
-
-    CHROMA_CU_S_420(4, 4)
+    CHROMA_CU_420(4, 4)
     CHROMA_CU_420(8, 8)
     CHROMA_CU_420(16, 16)
     CHROMA_CU_420(32, 32)
 
 
     p.chromaX265_CSP_I420.cuBLOCK_8x8.sa8d   = p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd;
-    p.chromaX265_CSP_I420.cuBLOCK_16x16.sa8d = sa8d8<8, 8>;
-    p.chromaX265_CSP_I420.cuBLOCK_32x32.sa8d = sa8d16<16, 16>;
-    p.chromaX265_CSP_I420.cuBLOCK_64x64.sa8d = sa8d16<32, 32>;
+    p.chromaX265_CSP_I420.cuBLOCK_16x16.sa8d = sa8d8_neon<8, 8>;
+    p.chromaX265_CSP_I420.cuBLOCK_32x32.sa8d = sa8d16_neon<16, 16>;
+    p.chromaX265_CSP_I420.cuBLOCK_64x64.sa8d = sa8d16_neon<32, 32>;
 
 
 #define CHROMA_PU_422(W, H) \
@@ -1986,90 +2427,97 @@
     p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
 
 
-    CHROMA_PU_422(4, 8);
-    CHROMA_PU_422(8, 16);
-    CHROMA_PU_422(16, 32);
-    CHROMA_PU_422(32, 64);
-    CHROMA_PU_422(4, 4);
+    CHROMA_PU_422(2, 4);
     CHROMA_PU_422(2, 8);
-    CHROMA_PU_422(8, 8);
+    CHROMA_PU_422(2, 16);
+    CHROMA_PU_422(4, 4);
+    CHROMA_PU_422(4, 8);
     CHROMA_PU_422(4, 16);
-    CHROMA_PU_422(8, 12);
-    CHROMA_PU_422(6, 16);
+    CHROMA_PU_422(4, 32);
     CHROMA_PU_422(8, 4);
-    CHROMA_PU_422(2, 16);
-    CHROMA_PU_422(16, 16);
+    CHROMA_PU_422(8, 8);
+    CHROMA_PU_422(8, 12);
+    CHROMA_PU_422(8, 16);
     CHROMA_PU_422(8, 32);
-    CHROMA_PU_422(16, 24);
+    CHROMA_PU_422(8, 64);
+    CHROMA_PU_422(6, 16);
     CHROMA_PU_422(12, 32);
     CHROMA_PU_422(16, 8);
-    CHROMA_PU_422(4,  32);
-    CHROMA_PU_422(32, 32);
+    CHROMA_PU_422(16, 16);
+    CHROMA_PU_422(16, 24);
+    CHROMA_PU_422(16, 32);
     CHROMA_PU_422(16, 64);
-    CHROMA_PU_422(32, 48);
     CHROMA_PU_422(24, 64);
     CHROMA_PU_422(32, 16);
-    CHROMA_PU_422(8,  64);
+    CHROMA_PU_422(32, 32);
+    CHROMA_PU_422(32, 48);
+    CHROMA_PU_422(32, 64);
 
 
     p.chromaX265_CSP_I422.puCHROMA_422_2x4.satd   = NULL;
-    p.chromaX265_CSP_I422.puCHROMA_422_8x16.satd  = satd8_neon<8, 16>;
-    p.chromaX265_CSP_I422.puCHROMA_422_16x32.satd = satd8_neon<16, 32>;
-    p.chromaX265_CSP_I422.puCHROMA_422_32x64.satd = satd8_neon<32, 64>;
-    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd   = pixel_satd_4x4_neon;
     p.chromaX265_CSP_I422.puCHROMA_422_2x8.satd   = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_2x16.satd  = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd   = satd4_neon<4, 4>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd   = satd4_neon<4, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x16.satd  = satd4_neon<4, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd  = satd4_neon<4, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_6x16.satd  = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd   = satd8_neon<8, 4>;
     p.chromaX265_CSP_I422.puCHROMA_422_8x8.satd   = satd8_neon<8, 8>;
-    p.chromaX265_CSP_I422.puCHROMA_422_16x16.satd = satd8_neon<16, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd  = satd8_neon<8, 12>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.satd  = satd8_neon<8, 16>;
     p.chromaX265_CSP_I422.puCHROMA_422_8x32.satd  = satd8_neon<8, 32>;
-    p.chromaX265_CSP_I422.puCHROMA_422_32x32.satd = satd8_neon<32, 32>;
-    p.chromaX265_CSP_I422.puCHROMA_422_16x64.satd = satd8_neon<16, 64>;
-    p.chromaX265_CSP_I422.puCHROMA_422_6x16.satd  = NULL;
-    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd   = satd4_neon<8, 4>;
-    p.chromaX265_CSP_I422.puCHROMA_422_2x16.satd  = NULL;
-    p.chromaX265_CSP_I422.puCHROMA_422_16x8.satd  = satd8_neon<16, 8>;
-    p.chromaX265_CSP_I422.puCHROMA_422_32x16.satd = satd8_neon<32, 16>;
-    
-    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd  = satd4_neon<8, 12>;
     p.chromaX265_CSP_I422.puCHROMA_422_8x64.satd  = satd8_neon<8, 64>;
     p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd = satd4_neon<12, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.satd  = satd8_neon<16, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.satd = satd8_neon<16, 16>;
     p.chromaX265_CSP_I422.puCHROMA_422_16x24.satd = satd8_neon<16, 24>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.satd = satd8_neon<16, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.satd = satd8_neon<16, 64>;
     p.chromaX265_CSP_I422.puCHROMA_422_24x64.satd = satd8_neon<24, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.satd = satd8_neon<32, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.satd = satd8_neon<32, 32>;
     p.chromaX265_CSP_I422.puCHROMA_422_32x48.satd = satd8_neon<32, 48>;
-
-#if HIGH_BIT_DEPTH
-    p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd   = satd4_neon<4, 8>;
-    p.chromaX265_CSP_I422.puCHROMA_422_4x16.satd  = satd4_neon<4, 16>;
-    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd  = satd4_neon<4, 32>;
-#endif // HIGH_BIT_DEPTH
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.satd = satd8_neon<32, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd = satd4_neon<12, 32>;
 
 
+#if HIGH_BIT_DEPTH
 #define CHROMA_CU_422(W, H) \
-    p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.sse_pp  = sse_neon<W, H, pixel, pixel>;  \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
-    p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_ps = blockcopy_ps_neon<W, H>; \
+    p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_ss = blockcopy_ss_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.sub_ps = pixel_sub_ps_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.add_psNONALIGNED = pixel_add_ps_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>;
-
-#define CHROMA_CU_S_422(W, H) \
+#else // !HIGH_BIT_DEPTH
+#define CHROMA_CU_422(W, H) \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_pp = blockcopy_pp_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_ps = blockcopy_ps_neon<W, H>; \
+    p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_ss = blockcopy_ss_neon<W, H>; \
+    p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.copy_sp = blockcopy_sp_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.sub_ps = pixel_sub_ps_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.add_psNONALIGNED = pixel_add_ps_neon<W, H>; \
     p.chromaX265_CSP_I422.cuBLOCK_422_ ## W ## x ## H.add_psALIGNED = pixel_add_ps_neon<W, H>;
-    
-    
-    CHROMA_CU_S_422(4, 8)
+#endif // HIGH_BIT_DEPTH
+
+
+    CHROMA_CU_422(4, 8)
     CHROMA_CU_422(8, 16)
     CHROMA_CU_422(16, 32)
     CHROMA_CU_422(32, 64)
 
-    p.chromaX265_CSP_I422.cuBLOCK_8x8.sa8d   = p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd;
-    p.chromaX265_CSP_I422.cuBLOCK_16x16.sa8d = sa8d8<8, 16>;
-    p.chromaX265_CSP_I422.cuBLOCK_32x32.sa8d = sa8d16<16, 32>;
-    p.chromaX265_CSP_I422.cuBLOCK_64x64.sa8d = sa8d16<32, 64>;
+    p.chromaX265_CSP_I422.cuBLOCK_8x8.sa8d       = p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd;
+    p.chromaX265_CSP_I422.cuBLOCK_16x16.sa8d     = sa8d8_neon<8, 16>;
+    p.chromaX265_CSP_I422.cuBLOCK_32x32.sa8d     = sa8d16_neon<16, 32>;
+    p.chromaX265_CSP_I422.cuBLOCK_64x64.sa8d     = sa8d16_neon<32, 64>;
+
+    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.sa8d  = sa8d8_neon<8, 16>;
+    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.sa8d = sa8d16_neon<16, 32>;
+    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sa8d = sa8d16_neon<32, 64>;
 
+    p.weight_pp = weight_pp_neon;
 
+    p.planecopy_cp = planecopy_cp_neon;
 }

x265-4.1.tar/source/common/aarch64/pixel-prim.h -> x265-4.2.tar/source/common/aarch64/pixel-prim.h Changed

x265-4.1.tar/source/common/aarch64/pixel-util-common.S -> x265-4.2.tar/source/common/aarch64/pixel-util-common.S Changed

x265-4.1.tar/source/common/aarch64/pixel-util-sve.S -> x265-4.2.tar/source/common/aarch64/pixel-util-sve.S Changed

@@ -56,261 +56,3 @@
     ret
 endfunc
 
-//******* satd *******
-.macro satd_4x4_sve
-    ld1b            {z0.h}, p0/z, x0
-    ld1b            {z2.h}, p0/z, x2
-    add             x0, x0, x1
-    add             x2, x2, x3
-    ld1b            {z1.h}, p0/z, x0
-    ld1b            {z3.h}, p0/z, x2
-    add             x0, x0, x1
-    add             x2, x2, x3
-    ld1b            {z4.h}, p0/z, x0
-    ld1b            {z6.h}, p0/z, x2
-    add             x0, x0, x1
-    add             x2, x2, x3
-    ld1b            {z5.h}, p0/z, x0
-    ld1b            {z7.h}, p0/z, x2
-    add             x0, x0, x1
-    add             x2, x2, x3
-
-    sub             z0.h, z0.h, z2.h
-    sub             z1.h, z1.h, z3.h
-    sub             z2.h, z4.h, z6.h
-    sub             z3.h, z5.h, z7.h
-
-    add             z4.h, z0.h, z2.h
-    add             z5.h, z1.h, z3.h
-    sub             z6.h, z0.h, z2.h
-    sub             z7.h, z1.h, z3.h
-
-    add             z0.h, z4.h, z5.h
-    sub             z1.h, z4.h, z5.h
-
-    add             z2.h, z6.h, z7.h
-    sub             z3.h, z6.h, z7.h
-
-    trn1            z4.h, z0.h, z2.h
-    trn2            z5.h, z0.h, z2.h
-
-    trn1            z6.h, z1.h, z3.h
-    trn2            z7.h, z1.h, z3.h
-
-    add             z0.h, z4.h, z5.h
-    sub             z1.h, z4.h, z5.h
-
-    add             z2.h, z6.h, z7.h
-    sub             z3.h, z6.h, z7.h
-
-    trn1            z4.s, z0.s, z1.s
-    trn2            z5.s, z0.s, z1.s
-
-    trn1            z6.s, z2.s, z3.s
-    trn2            z7.s, z2.s, z3.s
-
-    abs             z4.h, p0/m, z4.h
-    abs             z5.h, p0/m, z5.h
-    abs             z6.h, p0/m, z6.h
-    abs             z7.h, p0/m, z7.h
-
-    smax            z4.h, p0/m, z4.h, z5.h
-    smax            z6.h, p0/m, z6.h, z7.h
-
-    add             z0.h, z4.h, z6.h
-
-    uaddlp          v0.2s, v0.4h
-    uaddlp          v0.1d, v0.2s
-.endm
-
-// int satd_4x4(const pixel* pix1, intptr_t stride_pix1, const pixel* pix2, intptr_t stride_pix2)
-function PFX(pixel_satd_4x4_sve)
-    ptrue           p0.h, vl4
-    satd_4x4_sve
-    fmov            x0, d0
-    ret
-endfunc
-
-function PFX(pixel_satd_8x4_sve)
-    ptrue           p0.h, vl4
-    mov             x4, x0
-    mov             x5, x2
-    satd_4x4_sve
-    add             x0, x4, #4
-    add             x2, x5, #4
-    umov            x6, v0.d0
-    satd_4x4_sve
-    umov            x0, v0.d0
-    add             x0, x0, x6
-    ret
-endfunc
-
-function PFX(pixel_satd_8x12_sve)
-    ptrue           p0.h, vl4
-    mov             x4, x0
-    mov             x5, x2
-    mov             x7, #0
-    satd_4x4_sve
-    umov            x6, v0.d0
-    add             x7, x7, x6
-    add             x0, x4, #4
-    add             x2, x5, #4
-    satd_4x4_sve
-    umov            x6, v0.d0
-    add             x7, x7, x6
-.rept 2
-    sub             x0, x0, #4
-    sub             x2, x2, #4
-    mov             x4, x0
-    mov             x5, x2
-    satd_4x4_sve
-    umov            x6, v0.d0
-    add             x7, x7, x6
-    add             x0, x4, #4
-    add             x2, x5, #4
-    satd_4x4_sve
-    umov            x6, v0.d0
-    add             x7, x7, x6
-.endr
-    mov             x0, x7
-    ret
-endfunc
-
-.macro LOAD_DIFF_16x4_sve v0 v1 v2 v3 v4 v5 v6 v7
-    mov             x11, #8 // in order to consider CPUs whose vector size is greater than 128 bits
-    ld1b            {z0.h}, p0/z, x0
-    ld1b            {z1.h}, p0/z, x0, x11
-    ld1b            {z2.h}, p0/z, x2
-    ld1b            {z3.h}, p0/z, x2, x11
-    add             x0, x0, x1
-    add             x2, x2, x3
-    ld1b            {z4.h}, p0/z, x0
-    ld1b            {z5.h}, p0/z, x0, x11
-    ld1b            {z6.h}, p0/z, x2
-    ld1b            {z7.h}, p0/z, x2, x11
-    add             x0, x0, x1
-    add             x2, x2, x3
-    sub             \v0\().h, z0.h, z2.h
-    sub             \v4\().h, z1.h, z3.h
-    sub             \v1\().h, z4.h, z6.h
-    sub             \v5\().h, z5.h, z7.h
-
-    ld1b            {z0.h}, p0/z, x0
-    ld1b            {z1.h}, p0/z, x0, x11
-    ld1b            {z2.h}, p0/z, x2
-    ld1b            {z3.h}, p0/z, x2, x11
-    add             x0, x0, x1
-    add             x2, x2, x3
-    ld1b            {z4.h}, p0/z, x0
-    ld1b            {z5.h}, p0/z, x0, x11
-    ld1b            {z6.h}, p0/z, x2
-    ld1b            {z7.h}, p0/z, x2, x11
-    add             x0, x0, x1
-    add             x2, x2, x3
-    sub             \v2\().h, z0.h, z2.h
-    sub             \v6\().h, z1.h, z3.h
-    sub             \v3\().h, z4.h, z6.h
-    sub             \v7\().h, z5.h, z7.h
-.endm
-
-// one vertical hadamard pass and two horizontal
-function PFX(satd_8x4v_8x8h_sve), export=0
-    HADAMARD4_V     z16.h, z18.h, z17.h, z19.h, z0.h, z2.h, z1.h, z3.h
-    HADAMARD4_V     z20.h, z21.h, z22.h, z23.h, z0.h, z1.h, z2.h, z3.h
-    trn4            z0.h, z1.h, z2.h, z3.h, z16.h, z17.h, z18.h, z19.h
-    trn4            z4.h, z5.h, z6.h, z7.h, z20.h, z21.h, z22.h, z23.h
-    SUMSUB_ABCD     z16.h, z17.h, z18.h, z19.h, z0.h, z1.h, z2.h, z3.h
-    SUMSUB_ABCD     z20.h, z21.h, z22.h, z23.h, z4.h, z5.h, z6.h, z7.h
-    trn4            z0.s, z2.s, z1.s, z3.s, z16.s, z18.s, z17.s, z19.s
-    trn4            z4.s, z6.s, z5.s, z7.s, z20.s, z22.s, z21.s, z23.s
-    ABS8_SVE        z0.h, z1.h, z2.h, z3.h, z4.h, z5.h, z6.h, z7.h, p0
-    smax            z0.h, p0/m, z0.h, z2.h
-    smax            z1.h, p0/m, z1.h, z3.h
-    smax            z4.h, p0/m, z4.h, z6.h
-    smax            z5.h, p0/m, z5.h, z7.h
-    ret
-endfunc
-
-function PFX(satd_16x4_sve), export=0
-    LOAD_DIFF_16x4_sve  z16, z17, z18, z19, z20, z21, z22, z23
-    b                    PFX(satd_8x4v_8x8h_sve)
-endfunc
-
-.macro pixel_satd_32x8_sve
-    mov             x4, x0
-    mov             x5, x2
-.rept 2
-    bl              PFX(satd_16x4_sve)
-    add             z30.h, z30.h, z0.h
-    add             z31.h, z31.h, z1.h
-    add             z30.h, z30.h, z4.h
-    add             z31.h, z31.h, z5.h
-.endr
-    add             x0, x4, #16
-    add             x2, x5, #16
-.rept 2
-    bl              PFX(satd_16x4_sve)
-    add             z30.h, z30.h, z0.h
-    add             z31.h, z31.h, z1.h
-    add             z30.h, z30.h, z4.h
-    add             z31.h, z31.h, z5.h
-.endr
-.endm
-
-.macro satd_32x16_sve
-    movi            v30.2d, #0
-    movi            v31.2d, #0
-    pixel_satd_32x8_sve
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    pixel_satd_32x8_sve
-    add             z0.h, z30.h, z31.h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-.endm
-
-function PFX(pixel_satd_32x16_sve)
-    ptrue           p0.h, vl8
-    mov             x10, x30
-    satd_32x16_sve
-    mov             x0, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x32_sve)
-    ptrue           p0.h, vl8
-    mov             x10, x30
-    mov             x7, #0
-    satd_32x16_sve
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    add             x7, x7, x6
-    satd_32x16_sve
-    add             x0, x7, x6
-    ret             x10
-endfunc
-
-.macro satd_64x16_sve
-    mov             x8, x0
-    mov             x9, x2
-    satd_32x16_sve
-    add             x7, x7, x6
-    add             x0, x8, #32
-    add             x2, x9, #32
-    satd_32x16_sve
-    add             x7, x7, x6
-.endm
-
-function PFX(pixel_satd_64x48_sve)
-    ptrue           p0.h, vl8
-    mov             x10, x30
-    mov             x7, #0
-.rept 2
-    satd_64x16_sve
-    sub             x0, x0, #48
-    sub             x2, x2, #48
-.endr
-    satd_64x16_sve
-    mov             x0, x7
-    ret             x10
-endfunc

x265-4.2.tar/source/common/aarch64/pixel-util-sve2-bitperm.S Added

@@ -0,0 +1,125 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: George Steed <george.steed@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm-sve.S"
+#include "pixel-util-common.S"
+
+.arch armv8-a+sve2+sve2-bitperm
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+
+.text
+
+// int scanPosLast(
+//     const uint16_t *scan,      // x0
+//     const coeff_t *coeff,      // x1
+//     uint16_t *coeffSign,       // x2
+//     uint16_t *coeffFlag,       // x3
+//     uint8_t *coeffNum,         // x4
+//     int numSig,                // x5
+//     const uint16_t* scanCG4x4, // x6
+//     const int trSize)          // x7
+function PFX(scanPosLast_sve2_bitperm)
+    // Convert unit of trSize stride from elements (int16) to bytes.
+    add             x7, x7, x7
+
+    // Load scan table and convert to bytes.
+    ldp             q0, q1, x6
+    uzp1            v0.16b, v0.16b, v1.16b  // v0 - Zigzag scan table.
+
+    movrel          x10, g_SPL_and_mask
+    ldr             q28, x10              // v28 = mask for pmovmskb.
+    add             x10, x7, x7             // 2*x7
+    add             x11, x7, x7, lsl #1     // 3*x7
+    add             x9, x4, #1              // CG count
+
+1:
+    // Position of current CG.
+    ldrh            w6, x0, #32
+    add             x6, x1, x6, lsl #1
+
+    // Loading current CG and saturate to bytes.
+    ldr             d2, x6
+    ldr             d3, x6, x7
+    ldr             d4, x6, x10
+    ldr             d5, x6, x11
+    mov             v2.d1, v3.d0
+    mov             v4.d1, v5.d0
+    sqxtn           v2.8b, v2.8h
+    sqxtn2          v2.16b, v4.8h
+
+    // Apply zigzag.
+    tbl             v3.16b, {v2.16b}, v0.16b
+
+    // Get zero/sign.
+    cmeq            v5.16b, v3.16b, #0   // v5 = zero
+    cmlt            v3.16b, v3.16b, #0   // v3 = negative
+
+    //  val: v3.h0 = pmovmskb(v3).
+    // mask: v3.h1 = pmovmskb(v4).
+    and             v3.16b, v3.16b, v28.16b
+    bic             v4.16b, v28.16b, v5.16b
+    addp            v3.16b, v3.16b, v4.16b
+    addp            v3.16b, v3.16b, v3.16b
+    addp            v3.16b, v3.16b, v3.16b
+    fmov            w15, s3
+
+    // coeffNum = addv(v3 != 0) = 16 - addv(v5).
+    addv            b5, v5.16b
+    smov            w6, v5.b0
+    add             w6, w6, #16
+    sub             x5, x5, x6
+    strb            w6, x4, #1
+
+    // coeffFlag = reverse_bit(w15) in 16-bit.
+    rbit            w12, w15
+    strh            w12, x3, #2
+
+    // Pack bits from z3.h0 into z30.h0, based on z3.h1 mask.
+    mov             h31, v3.h1
+    bext            z30.h, z3.h, z31.h
+    str             h30, x2, #2
+
+    cbnz            x5, 1b
+
+    // Count trailing zeros in (reversed) coeffFlag.
+    clz             w13, w15
+    lsr             w12, w12, w13
+    strh            w12, x3, #-2
+
+    // Get last pos.
+    sub             x9, x4, x9
+    eor             w13, w13, #15
+    add             x0, x13, x9, lsl #4
+    ret
+endfunc
+
+const g_SPL_and_mask, align=8
+.byte 0x1, 0x2, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80, 0x1, 0x2, 0x4, 0x8, 0x10, 0x20, 0x40, 0x80
+endconst

x265-4.1.tar/source/common/aarch64/pixel-util-sve2.S -> x265-4.2.tar/source/common/aarch64/pixel-util-sve2.S Changed

@@ -36,200 +36,6 @@
 
 .text
 
-// uint64_t pixel_var(const pixel* pix, intptr_t i_stride)
-function PFX(pixel_var_8x8_sve2)
-    ptrue           p0.h, vl8
-    ld1b            {z0.h}, p0/z, x0
-    add             x0, x0, x1
-    mul             z31.h, z0.h, z0.h
-    uaddlp          v1.4s, v31.8h
-.rept 7
-    ld1b            {z4.h}, p0/z, x0
-    add             x0, x0, x1
-    add             z0.h, z0.h, z4.h
-    mul             z31.h, z4.h, z4.h
-    uadalp          z1.s, p0/m, z31.h
-.endr
-    uaddlv          s0, v0.8h
-    uaddlv          d1, v1.4s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-endfunc
-
-function PFX(pixel_var_16x16_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_var_16x16
-    pixel_var_start
-    mov             w12, #16
-.Loop_var_16_sve2:
-    sub             w12, w12, #1
-    ld1             {v4.16b}, x0, x1
-    pixel_var_1 v4
-    cbnz            w12, .Loop_var_16_sve2
-    pixel_var_end
-    ret
-.vl_gt_16_pixel_var_16x16:
-    ptrue           p0.h, vl16
-    mov             z0.d, #0
-.rept 16
-    ld1b            {z4.h}, p0/z, x0
-    add             x0, x0, x1
-    add             z0.h, z0.h, z4.h
-    mul             z30.h, z4.h, z4.h
-    uadalp          z1.s, p0/m, z30.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z1.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-endfunc
-
-function PFX(pixel_var_32x32_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_var_32x32
-    pixel_var_start
-    mov             w12, #32
-.Loop_var_32_sve2:
-    sub             w12, w12, #1
-    ld1             {v4.16b-v5.16b}, x0, x1
-    pixel_var_1 v4
-    pixel_var_1 v5
-    cbnz            w12, .Loop_var_32_sve2
-    pixel_var_end
-    ret
-.vl_gt_16_pixel_var_32x32:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_var_32x32
-    ptrue           p0.b, vl32
-    mov             z0.d, #0
-    mov             z1.d, #0
-.rept 32
-    ld1b            {z4.b}, p0/z, x0
-    add             x0, x0, x1
-    uaddwb          z0.h, z0.h, z4.b
-    uaddwt          z0.h, z0.h, z4.b
-    umullb          z28.h, z4.b, z4.b
-    umullt          z29.h, z4.b, z4.b
-    uadalp          z1.s, p0/m, z28.h
-    uadalp          z1.s, p0/m, z29.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z1.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-.vl_gt_48_pixel_var_32x32:
-    ptrue           p0.h, vl32
-    mov             z0.d, #0
-    mov             z1.d, #0
-.rept 32
-    ld1b            {z4.h}, p0/z, x0
-    add             x0, x0, x1
-    add             z0.h, z0.h, z4.h
-    mul             z28.h, z4.h, z4.h
-    uadalp          z1.s, p0/m, z28.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z1.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-endfunc
-
-function PFX(pixel_var_64x64_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_var_64x64
-    pixel_var_start
-    mov             w12, #64
-.Loop_var_64_sve2:
-    sub             w12, w12, #1
-    ld1             {v4.16b-v7.16b}, x0, x1
-    pixel_var_1 v4
-    pixel_var_1 v5
-    pixel_var_1 v6
-    pixel_var_1 v7
-    cbnz            w12, .Loop_var_64_sve2
-    pixel_var_end
-    ret
-.vl_gt_16_pixel_var_64x64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_var_64x64
-    ptrue           p0.b, vl32
-    mov             z0.d, #0
-    mov             z2.d, #0
-.rept 64
-    ld1b            {z4.b}, p0/z, x0
-    ld1b            {z5.b}, p0/z, x0, #1, mul vl
-    add             x0, x0, x1
-    uaddwb          z0.h, z0.h, z4.b
-    uaddwt          z0.h, z0.h, z4.b
-    uaddwb          z0.h, z0.h, z5.b
-    uaddwt          z0.h, z0.h, z5.b
-    umullb          z24.h, z4.b, z4.b
-    umullt          z25.h, z4.b, z4.b
-    umullb          z26.h, z5.b, z5.b
-    umullt          z27.h, z5.b, z5.b
-    uadalp          z2.s, p0/m, z24.h
-    uadalp          z2.s, p0/m, z25.h
-    uadalp          z2.s, p0/m, z26.h
-    uadalp          z2.s, p0/m, z27.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z2.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-.vl_gt_48_pixel_var_64x64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_pixel_var_64x64
-    ptrue           p0.b, vl64
-    mov             z0.d, #0
-    mov             z1.d, #0
-.rept 64
-    ld1b            {z4.b}, p0/z, x0
-    add             x0, x0, x1
-    uaddwb          z0.h, z0.h, z4.b
-    uaddwt          z0.h, z0.h, z4.b
-    umullb          z24.h, z4.b, z4.b
-    umullt          z25.h, z4.b, z4.b
-    uadalp          z2.s, p0/m, z24.h
-    uadalp          z2.s, p0/m, z25.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z2.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-.vl_gt_112_pixel_var_64x64:
-    ptrue           p0.h, vl64
-    mov             z0.d, #0
-    mov             z1.d, #0
-.rept 64
-    ld1b            {z4.h}, p0/z, x0
-    add             x0, x0, x1
-    add             z0.h, z0.h, z4.h
-    mul             z24.h, z4.h, z4.h
-    uadalp          z1.s, p0/m, z24.h
-.endr
-    uaddv           d0, p0, z0.h
-    uaddv           d1, p0, z1.s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32
-    ret
-endfunc
-
 function PFX(getResidual16_sve2)
     rdvl            x9, #1
     cmp             x9, #16
@@ -407,8 +213,8 @@
     ret
 .vl_gt_16_pixel_sub_ps_64x64:
     rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_sub_ps_64x64
+    cmp             x9, #32
+    bgt             .vl_gt_48_pixel_sub_ps_64x64
     ptrue           p0.b, vl32
     mov             w12, #16
 .vl_gt_16_loop_sub_ps_64_sve2:
@@ -530,292 +336,6 @@
     ret
 endfunc
 
-function PFX(pixel_add_ps_4x4_sve2)
-    ptrue           p0.h, vl8
-    ptrue           p1.h, vl4
-.rept 4
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z2.h}, p1/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z4.h, z0.h, z2.h
-    sqxtunb         z4.b, z4.h
-    st1b            {z4.h}, p1, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(pixel_add_ps_8x8_sve2)
-    ptrue           p0.h, vl8
-.rept 8
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z2.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z4.h, z0.h, z2.h
-    sqxtunb         z4.b, z4.h
-    st1b            {z4.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-.macro pixel_add_ps_16xN_sve2 h
-function PFX(pixel_add_ps_16x\h\()_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_add_ps_16x\h
-    ptrue           p0.b, vl16
-.rept \h
-    ld1b            {z0.h}, p0/z, x2
-    ld1b            {z1.h}, p0/z, x2, #1, mul vl
-    ld1h            {z2.h}, p0/z, x3
-    ld1h            {z3.h}, p0/z, x3, #1, mul vl
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z2.h
-    add             z25.h, z1.h, z3.h
-    sqxtunb         z6.b, z24.h
-    sqxtunb         z7.b, z25.h
-    st1b            {z6.h}, p0, x0
-    st1b            {z7.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_16_pixel_add_ps_16x\h\():
-    ptrue           p0.b, vl32
-.rept \h
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z2.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z2.h
-    sqxtunb         z6.b, z24.h
-    st1b            {z6.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-.endm
-
-pixel_add_ps_16xN_sve2 16
-pixel_add_ps_16xN_sve2 32
-
-.macro pixel_add_ps_32xN_sve2 h
- function PFX(pixel_add_ps_32x\h\()_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_add_ps_32x\h
-    lsl             x5, x5, #1
-    mov             w12, #\h / 4
-.Loop_add_ps__sve2_32x\h\():
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.16b-v1.16b}, x2, x4
-    ld1             {v16.8h-v19.8h}, x3, x5
-    uxtl            v4.8h, v0.8b
-    uxtl2           v5.8h, v0.16b
-    uxtl            v6.8h, v1.8b
-    uxtl2           v7.8h, v1.16b
-    add             v24.8h, v4.8h, v16.8h
-    add             v25.8h, v5.8h, v17.8h
-    add             v26.8h, v6.8h, v18.8h
-    add             v27.8h, v7.8h, v19.8h
-    sqxtun          v4.8b, v24.8h
-    sqxtun2         v4.16b, v25.8h
-    sqxtun          v5.8b, v26.8h
-    sqxtun2         v5.16b, v27.8h
-    st1             {v4.16b-v5.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_add_ps__sve2_32x\h
-    ret
-.vl_gt_16_pixel_add_ps_32x\h\():
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_add_ps_32x\h
-    ptrue           p0.b, vl32
-.rept \h
-    ld1b            {z0.h}, p0/z, x2
-    ld1b            {z1.h}, p0/z, x2, #1, mul vl
-    ld1h            {z4.h}, p0/z, x3
-    ld1h            {z5.h}, p0/z, x3, #1, mul vl
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z4.h
-    add             z25.h, z1.h, z5.h
-    sqxtunb         z6.b, z24.h
-    sqxtunb         z7.b, z25.h
-    st1b            {z6.h}, p0, x0
-    st1b            {z7.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_48_pixel_add_ps_32x\h\():
-    ptrue           p0.b, vl64
-.rept \h
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z4.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z4.h
-    sqxtunb         z6.b, z24.h
-    st1b            {z6.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-.endm
-
-pixel_add_ps_32xN_sve2 32
-pixel_add_ps_32xN_sve2 64
-
-function PFX(pixel_add_ps_64x64_sve2)
-    rdvl            x9, #1
-    cmp             x9, #16
-    bgt             .vl_gt_16_pixel_add_ps_64x64
-    ptrue           p0.b, vl16
-.rept 64
-    ld1b            {z0.h}, p0/z, x2
-    ld1b            {z1.h}, p0/z, x2, #1, mul vl
-    ld1b            {z2.h}, p0/z, x2, #2, mul vl
-    ld1b            {z3.h}, p0/z, x2, #3, mul vl
-    ld1b            {z4.h}, p0/z, x2, #4 ,mul vl
-    ld1b            {z5.h}, p0/z, x2, #5, mul vl
-    ld1b            {z6.h}, p0/z, x2, #6, mul vl
-    ld1b            {z7.h}, p0/z, x2, #7, mul vl
-    ld1h            {z8.h}, p0/z, x3
-    ld1h            {z9.h}, p0/z, x3, #1, mul vl
-    ld1h            {z10.h}, p0/z, x3, #2, mul vl
-    ld1h            {z11.h}, p0/z, x3, #3, mul vl
-    ld1h            {z12.h}, p0/z, x3, #4, mul vl
-    ld1h            {z13.h}, p0/z, x3, #5, mul vl
-    ld1h            {z14.h}, p0/z, x3, #6, mul vl
-    ld1h            {z15.h}, p0/z, x3, #7, mul vl
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z8.h
-    add             z25.h, z1.h, z9.h
-    add             z26.h, z2.h, z10.h
-    add             z27.h, z3.h, z11.h
-    add             z28.h, z4.h, z12.h
-    add             z29.h, z5.h, z13.h
-    add             z30.h, z6.h, z14.h
-    add             z31.h, z7.h, z15.h
-    sqxtunb         z6.b, z24.h
-    sqxtunb         z7.b, z25.h
-    sqxtunb         z8.b, z26.h
-    sqxtunb         z9.b, z27.h
-    sqxtunb         z10.b, z28.h
-    sqxtunb         z11.b, z29.h
-    sqxtunb         z12.b, z30.h
-    sqxtunb         z13.b, z31.h
-    st1b            {z6.h}, p0, x0
-    st1b            {z7.h}, p0, x0, #1, mul vl
-    st1b            {z8.h}, p0, x0, #2, mul vl
-    st1b            {z9.h}, p0, x0, #3, mul vl
-    st1b            {z10.h}, p0, x0, #4, mul vl
-    st1b            {z11.h}, p0, x0, #5, mul vl
-    st1b            {z12.h}, p0, x0, #6, mul vl
-    st1b            {z13.h}, p0, x0, #7, mul vl
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_16_pixel_add_ps_64x64:
-    cmp             x9, #48
-    bgt             .vl_gt_48_pixel_add_ps_64x64
-    ptrue           p0.b, vl32
-.rept 64
-    ld1b            {z0.h}, p0/z, x2
-    ld1b            {z1.h}, p0/z, x2, #1, mul vl
-    ld1b            {z2.h}, p0/z, x2, #2, mul vl
-    ld1b            {z3.h}, p0/z, x2, #3, mul vl
-    ld1h            {z8.h}, p0/z, x3
-    ld1h            {z9.h}, p0/z, x3, #1, mul vl
-    ld1h            {z10.h}, p0/z, x3, #2, mul vl
-    ld1h            {z11.h}, p0/z, x3, #3, mul vl
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z8.h
-    add             z25.h, z1.h, z9.h
-    add             z26.h, z2.h, z10.h
-    add             z27.h, z3.h, z11.h
-    sqxtunb         z6.b, z24.h
-    sqxtunb         z7.b, z25.h
-    sqxtunb         z8.b, z26.h
-    sqxtunb         z9.b, z27.h
-    st1b            {z6.h}, p0, x0
-    st1b            {z7.h}, p0, x0, #1, mul vl
-    st1b            {z8.h}, p0, x0, #2, mul vl
-    st1b            {z9.h}, p0, x0, #3, mul vl
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_48_pixel_add_ps_64x64:
-    cmp             x9, #112
-    bgt             .vl_gt_112_pixel_add_ps_64x64
-    ptrue           p0.b, vl64
-.rept 64
-    ld1b            {z0.h}, p0/z, x2
-    ld1b            {z1.h}, p0/z, x2, #1, mul vl
-    ld1h            {z8.h}, p0/z, x3
-    ld1h            {z9.h}, p0/z, x3, #1, mul vl
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z8.h
-    add             z25.h, z1.h, z9.h
-    sqxtunb         z6.b, z24.h
-    sqxtunb         z7.b, z25.h
-    st1b            {z6.h}, p0, x0
-    st1b            {z7.h}, p0, x0, #1, mul vl
-    add             x0, x0, x1
-.endr
-    ret
-.vl_gt_112_pixel_add_ps_64x64:
-    ptrue           p0.b, vl128
-.rept 64
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z8.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z24.h, z0.h, z8.h
-    sqxtunb         z6.b, z24.h
-    st1b            {z6.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-// Chroma add_ps
-function PFX(pixel_add_ps_4x8_sve2)
-    ptrue           p0.h,vl4
-.rept 8
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z2.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z4.h, z0.h, z2.h
-    sqxtunb         z4.b, z4.h
-    st1b            {z4.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(pixel_add_ps_8x16_sve2)
-    ptrue           p0.h,vl8
-.rept 16
-    ld1b            {z0.h}, p0/z, x2
-    ld1h            {z2.h}, p0/z, x3
-    add             x2, x2, x4
-    add             x3, x3, x5, lsl #1
-    add             z4.h, z0.h, z2.h
-    sqxtunb         z4.b, z4.h
-    st1b            {z4.h}, p0, x0
-    add             x0, x0, x1
-.endr
-    ret
-endfunc
-
 // void scale1D_128to64(pixel *dst, const pixel *src)
 function PFX(scale1D_128to64_sve2)
     rdvl            x9, #1
@@ -866,88 +386,18 @@
     ret
 endfunc
 
-/***** dequant_scaling*****/
-// void dequant_scaling_c(const int16_t* quantCoef, const int32_t* deQuantCoef, int16_t* coef, int num, int per, int shift)
-function PFX(dequant_scaling_sve2)
-    ptrue           p0.h, vl8
-    add             x5, x5, #4              // shift + 4
-    lsr             x3, x3, #3              // num / 8
-    cmp             x5, x4
-    blt             .dequant_skip_sve2
-
-    mov             x12, #1
-    sub             x6, x5, x4          // shift - per
-    sub             x6, x6, #1          // shift - per - 1
-    lsl             x6, x12, x6         // 1 << shift - per - 1 (add)
-    mov             z0.s, w6
-    sub             x7, x4, x5          // per - shift
-    mov             z3.s, w7
-
-.dequant_loop1_sve2:
-    ld1h            {z19.h}, p0/z, x0
-    ld1w            {z2.s}, p0/z, x1
-    add             x1, x1, #16
-    ld1w            {z20.s}, p0/z, x1
-    add             x0, x0, #16
-    add             x1, x1, #16
-
-    sub             x3, x3, #1
-    sunpklo         z1.s, z19.h
-    sunpkhi         z19.s, z19.h
-
-    mul             z1.s, z1.s, z2.s // quantCoef * deQuantCoef
-    mul             z19.s, z19.s, z20.s
-    add             z1.s, z1.s, z0.s // quantCoef * deQuantCoef + add
-    add             z19.s, z19.s, z0.s
-
-    // No equivalent instructions in SVE2 for sshl
-    // as sqshl has double latency
-    sshl            v1.4s, v1.4s, v3.4s
-    sshl            v19.4s, v19.4s, v3.4s
-
-    sqxtnb          z16.h, z1.s
-    sqxtnb          z17.h, z19.s
-    st1h            {z16.s}, p0, x2
-    st1h            {z17.s}, p0, x2, #1, mul vl
-    add             x2, x2, #16
-    cbnz            x3, .dequant_loop1_sve2
-    ret
-
-.dequant_skip_sve2:
-    sub             x6, x4, x5          // per - shift
-    mov             z0.h, w6
-
-.dequant_loop2_sve2:
-    ld1h            {z19.h}, p0/z, x0
-    ld1w            {z2.s}, p0/z, x1
-    add             x1, x1, #16
-    ld1w            {z20.s}, p0/z, x1
-    add             x0, x0, #16
-    add             x1, x1, #16
-
-
-    sub             x3, x3, #1
-    sunpklo         z1.s, z19.h
-    sunpkhi         z19.s, z19.h
-
-    mul             z1.s, z1.s, z2.s // quantCoef * deQuantCoef
-    mul             z19.s, z19.s, z20.s
-
-    // Keeping NEON instructions here in order to have
-    // one sqshl later
-    sqxtn           v16.4h, v1.4s       // x265_clip3
-    sqxtn2          v16.8h, v19.4s
-
-    sqshl           z16.h, p0/m, z16.h, z0.h // coefQ << per - shift
-    st1h            {z16.h}, p0, x2
-    add             x2, x2, #16
-    cbnz            x3, .dequant_loop2_sve2
-    ret
-endfunc
-
 // void dequant_normal_c(const int16_t* quantCoef, int16_t* coef, int num, int scale, int shift)
 function PFX(dequant_normal_sve2)
-    lsr             w2, w2, #4              // num / 16
+//  X265_CHECK(num >= 4 * 4, "dequant num %d too small\n", num);
+//  X265_CHECK(num <= 32 * 32, "dequant num %d too large\n", num);
+//  X265_CHECK((num % 16) == 0, "dequant num %d not multiple of 16\n", num);
+#if HIGH_BIT_DEPTH
+    cmp             w3, #32768
+    blt             .dqn_skip
+    lsr             w3, w3, #(BIT_DEPTH - 8)
+    sub             w4, w4, #(BIT_DEPTH - 8)
+.dqn_skip:
+#endif
     neg             w4, w4
     mov             z0.h, w3
     mov             z1.s, w4
@@ -971,7 +421,7 @@
     sqxtn           v3.4h, v18.4s
     sqxtn2          v3.8h, v19.4s
 
-    sub             w2, w2, #1
+    sub             w2, w2, #16
     st1             {v2.8h, v3.8h}, x1, #32
     cbnz            w2, .dqn_loop1_sve2
     ret
@@ -988,8 +438,8 @@
 
     sqxtnb          z2.h, z16.s
     sqxtnt          z2.h, z17.s
-    
-    sub             w2, w2, #1
+
+    sub             w2, w2, #16
     st1h            {z2.h}, p0, x1
     add             x1, x1, #32
     cbnz            w2, .gt_16_dqn_loop1_sve2
@@ -1417,12 +867,13 @@
 .vl_gt_112_ssimDist64:
     ssimDist_start_sve2
     ptrue           p0.s, vl32
+    mov             x5, #32
 .vl_gt_112_loop_ssimDist64_sve2:
     sub             w12, w12, #1
     ld1b            {z2.s}, p0/z, x0
-    ld1b            {z3.s}, p0/z, x0, #1, mul vl
+    ld1b            {z3.s}, p0/z, x0, x5
     ld1b            {z23.s}, p0/z, x2
-    ld1b            {z24.s}, p0/z, x2, #1, mul vl
+    ld1b            {z24.s}, p0/z, x2, x5
     ssimDist_1_sve2 z2, z3, z23, z24
     add             x0, x0, x1
     add             x2, x2, x3
@@ -1674,10 +1125,11 @@
 .vl_gt_112_normFact64:
     normFact_start_sve2
     ptrue           p0.s, vl32
+    mov             x4, #32
 .vl_gt_112_loop_normFact64_sve2:
     sub             w12, w12, #1
     ld1b            {z4.s}, p0/z, x0
-    ld1b            {z5.s}, p0/z, x0, #1, mul vl
+    ld1b            {z5.s}, p0/z, x0, x4
     normFact_1_sve2 z4, z5
     add             x0, x0, x1
     cbnz            w12, .vl_gt_112_loop_normFact64_sve2

x265-4.1.tar/source/common/aarch64/pixel-util.S -> x265-4.2.tar/source/common/aarch64/pixel-util.S Changed

@@ -36,67 +36,6 @@
 
 .text
 
-// uint64_t pixel_var(const pixel* pix, intptr_t i_stride)
-function PFX(pixel_var_8x8_neon)
-    ld1             {v4.8b}, x0, x1        // pixx
-    uxtl            v0.8h, v4.8b             // sum = pixx
-    umull           v1.8h, v4.8b, v4.8b
-    uaddlp          v1.4s, v1.8h             // sqr = pixx * pixx
-
-.rept 7
-    ld1             {v4.8b}, x0, x1        // pixx
-    umull           v31.8h, v4.8b, v4.8b
-    uaddw           v0.8h, v0.8h, v4.8b      // sum += pixx
-    uadalp          v1.4s, v31.8h            // sqr += pixx * pixx
-.endr
-    uaddlv          s0, v0.8h
-    uaddlv          d1, v1.4s
-    fmov            w0, s0
-    fmov            x1, d1
-    orr             x0, x0, x1, lsl #32      // return sum + ((uint64_t)sqr << 32);
-    ret
-endfunc
-
-function PFX(pixel_var_16x16_neon)
-    pixel_var_start
-    mov             w12, #16
-.Loop_var_16:
-    sub             w12, w12, #1
-    ld1             {v4.16b}, x0, x1
-    pixel_var_1 v4
-    cbnz            w12, .Loop_var_16
-    pixel_var_end
-    ret
-endfunc
-
-function PFX(pixel_var_32x32_neon)
-    pixel_var_start
-    mov             w12, #32
-.Loop_var_32:
-    sub             w12, w12, #1
-    ld1             {v4.16b-v5.16b}, x0, x1
-    pixel_var_1 v4
-    pixel_var_1 v5
-    cbnz            w12, .Loop_var_32
-    pixel_var_end
-    ret
-endfunc
-
-function PFX(pixel_var_64x64_neon)
-    pixel_var_start
-    mov             w12, #64
-.Loop_var_64:
-    sub             w12, w12, #1
-    ld1             {v4.16b-v7.16b}, x0, x1
-    pixel_var_1 v4
-    pixel_var_1 v5
-    pixel_var_1 v6
-    pixel_var_1 v7
-    cbnz            w12, .Loop_var_64
-    pixel_var_end
-    ret
-endfunc
-
 // void getResidual4_neon(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride)
 function PFX(getResidual4_neon)
     lsl             x4, x3, #1
@@ -340,189 +279,6 @@
     ret
 endfunc
 
-// void x265_pixel_add_ps_neon(pixel* a, intptr_t dstride, const pixel* b0, const int16_t* b1, intptr_t sstride0, intptr_t sstride1);
-function PFX(pixel_add_ps_4x4_neon)
-    lsl             x5, x5, #1
-.rept 2
-    ld1             {v0.8b}, x2, x4
-    ld1             {v1.8b}, x2, x4
-    ld1             {v2.4h}, x3, x5
-    ld1             {v3.4h}, x3, x5
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    add             v4.8h, v0.8h, v2.8h
-    add             v5.8h, v1.8h, v3.8h
-    sqxtun          v4.8b, v4.8h
-    sqxtun          v5.8b, v5.8h
-    st1             {v4.s}0, x0, x1
-    st1             {v5.s}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(pixel_add_ps_8x8_neon)
-    lsl             x5, x5, #1
-.rept 4
-    ld1             {v0.8b}, x2, x4
-    ld1             {v1.8b}, x2, x4
-    ld1             {v2.8h}, x3, x5
-    ld1             {v3.8h}, x3, x5
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    add             v4.8h, v0.8h, v2.8h
-    add             v5.8h, v1.8h, v3.8h
-    sqxtun          v4.8b, v4.8h
-    sqxtun          v5.8b, v5.8h
-    st1             {v4.8b}, x0, x1
-    st1             {v5.8b}, x0, x1
-.endr
-    ret
-endfunc
-
-.macro pixel_add_ps_16xN_neon h
-function PFX(pixel_add_ps_16x\h\()_neon)
-    lsl             x5, x5, #1
-    mov             w12, #\h / 8
-.Loop_add_ps_16x\h\():
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.16b}, x2, x4
-    ld1             {v1.16b}, x2, x4
-    ld1             {v16.8h-v17.8h}, x3, x5
-    ld1             {v18.8h-v19.8h}, x3, x5
-    uxtl            v4.8h, v0.8b
-    uxtl2           v5.8h, v0.16b
-    uxtl            v6.8h, v1.8b
-    uxtl2           v7.8h, v1.16b
-    add             v24.8h, v4.8h, v16.8h
-    add             v25.8h, v5.8h, v17.8h
-    add             v26.8h, v6.8h, v18.8h
-    add             v27.8h, v7.8h, v19.8h
-    sqxtun          v4.8b, v24.8h
-    sqxtun2         v4.16b, v25.8h
-    sqxtun          v5.8b, v26.8h
-    sqxtun2         v5.16b, v27.8h
-    st1             {v4.16b}, x0, x1
-    st1             {v5.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_add_ps_16x\h
-    ret
-endfunc
-.endm
-
-pixel_add_ps_16xN_neon 16
-pixel_add_ps_16xN_neon 32
-
-.macro pixel_add_ps_32xN_neon h
- function PFX(pixel_add_ps_32x\h\()_neon)
-    lsl             x5, x5, #1
-    mov             w12, #\h / 4
-.Loop_add_ps_32x\h\():
-    sub             w12, w12, #1
-.rept 4
-    ld1             {v0.16b-v1.16b}, x2, x4
-    ld1             {v16.8h-v19.8h}, x3, x5
-    uxtl            v4.8h, v0.8b
-    uxtl2           v5.8h, v0.16b
-    uxtl            v6.8h, v1.8b
-    uxtl2           v7.8h, v1.16b
-    add             v24.8h, v4.8h, v16.8h
-    add             v25.8h, v5.8h, v17.8h
-    add             v26.8h, v6.8h, v18.8h
-    add             v27.8h, v7.8h, v19.8h
-    sqxtun          v4.8b, v24.8h
-    sqxtun2         v4.16b, v25.8h
-    sqxtun          v5.8b, v26.8h
-    sqxtun2         v5.16b, v27.8h
-    st1             {v4.16b-v5.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_add_ps_32x\h
-    ret
-endfunc
-.endm
-
-pixel_add_ps_32xN_neon 32
-pixel_add_ps_32xN_neon 64
-
-function PFX(pixel_add_ps_64x64_neon)
-    lsl             x5, x5, #1
-    sub             x5, x5, #64
-    mov             w12, #32
-.Loop_add_ps_64x64:
-    sub             w12, w12, #1
-.rept 2
-    ld1             {v0.16b-v3.16b}, x2, x4
-    ld1             {v16.8h-v19.8h}, x3, #64
-    ld1             {v20.8h-v23.8h}, x3, x5
-    uxtl            v4.8h, v0.8b
-    uxtl2           v5.8h, v0.16b
-    uxtl            v6.8h, v1.8b
-    uxtl2           v7.8h, v1.16b
-    uxtl            v24.8h, v2.8b
-    uxtl2           v25.8h, v2.16b
-    uxtl            v26.8h, v3.8b
-    uxtl2           v27.8h, v3.16b
-    add             v0.8h, v4.8h, v16.8h
-    add             v1.8h, v5.8h, v17.8h
-    add             v2.8h, v6.8h, v18.8h
-    add             v3.8h, v7.8h, v19.8h
-    add             v4.8h, v24.8h, v20.8h
-    add             v5.8h, v25.8h, v21.8h
-    add             v6.8h, v26.8h, v22.8h
-    add             v7.8h, v27.8h, v23.8h
-    sqxtun          v0.8b, v0.8h
-    sqxtun2         v0.16b, v1.8h
-    sqxtun          v1.8b, v2.8h
-    sqxtun2         v1.16b, v3.8h
-    sqxtun          v2.8b, v4.8h
-    sqxtun2         v2.16b, v5.8h
-    sqxtun          v3.8b, v6.8h
-    sqxtun2         v3.16b, v7.8h
-    st1             {v0.16b-v3.16b}, x0, x1
-.endr
-    cbnz            w12, .Loop_add_ps_64x64
-    ret
-endfunc
-
-// Chroma add_ps
-function PFX(pixel_add_ps_4x8_neon)
-    lsl             x5, x5, #1
-.rept 4
-    ld1             {v0.8b}, x2, x4
-    ld1             {v1.8b}, x2, x4
-    ld1             {v2.4h}, x3, x5
-    ld1             {v3.4h}, x3, x5
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    add             v4.8h, v0.8h, v2.8h
-    add             v5.8h, v1.8h, v3.8h
-    sqxtun          v4.8b, v4.8h
-    sqxtun          v5.8b, v5.8h
-    st1             {v4.s}0, x0, x1
-    st1             {v5.s}0, x0, x1
-.endr
-    ret
-endfunc
-
-function PFX(pixel_add_ps_8x16_neon)
-    lsl             x5, x5, #1
-.rept 8
-    ld1             {v0.8b}, x2, x4
-    ld1             {v1.8b}, x2, x4
-    ld1             {v2.8h}, x3, x5
-    ld1             {v3.8h}, x3, x5
-    uxtl            v0.8h, v0.8b
-    uxtl            v1.8h, v1.8b
-    add             v4.8h, v0.8h, v2.8h
-    add             v5.8h, v1.8h, v3.8h
-    sqxtun          v4.8b, v4.8h
-    sqxtun          v5.8b, v5.8h
-    st1             {v4.8b}, x0, x1
-    st1             {v5.8b}, x0, x1
-.endr
-    ret
-endfunc
-
 // void scale1D_128to64(pixel *dst, const pixel *src)
 function PFX(scale1D_128to64_neon)
 .rept 2
@@ -565,1000 +321,6 @@
     ret
 endfunc
 
-// void planecopy_cp_c(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift)
-function PFX(pixel_planecopy_cp_neon)
-    dup             v2.16b, w6
-    sub             x5, x5, #1
-.Loop_h:
-    mov             x6, x0
-    mov             x12, x2
-    mov             x7, #0
-.Loop_w:
-    ldr             q0, x6, #16
-    ushl            v0.16b, v0.16b, v2.16b
-    str             q0, x12, #16
-    add             x7, x7, #16
-    cmp             x7, x4
-    blt             .Loop_w
-
-    add             x0, x0, x1
-    add             x2, x2, x3
-    sub             x5, x5, #1
-    cbnz            x5, .Loop_h
-
-// handle last row
-    mov             x5, x4
-    lsr             x5, x5, #3
-.LoopW8:
-    ldr             d0, x0, #8
-    ushl            v0.8b, v0.8b, v2.8b
-    str             d0, x2, #8
-    sub             x4, x4, #8
-    sub             x5, x5, #1
-    cbnz            x5, .LoopW8
-
-    mov             x5, #8
-    sub             x5, x5, x4
-    sub             x0, x0, x5
-    sub             x2, x2, x5
-    ldr             d0, x0
-    ushl            v0.8b, v0.8b, v2.8b
-    str             d0, x2
-    ret
-endfunc
-
-//******* satd *******
-.macro satd_4x4_neon
-    ld1             {v0.s}0, x0, x1
-    ld1             {v0.s}1, x0, x1
-    ld1             {v1.s}0, x2, x3
-    ld1             {v1.s}1, x2, x3
-    ld1             {v2.s}0, x0, x1
-    ld1             {v2.s}1, x0, x1
-    ld1             {v3.s}0, x2, x3
-    ld1             {v3.s}1, x2, x3
-
-    usubl           v4.8h, v0.8b, v1.8b
-    usubl           v5.8h, v2.8b, v3.8b
-
-    add             v6.8h, v4.8h, v5.8h
-    sub             v7.8h, v4.8h, v5.8h
-
-    mov             v4.d0, v6.d1
-    add             v0.4h, v6.4h, v4.4h
-    sub             v2.4h, v6.4h, v4.4h
-
-    mov             v5.d0, v7.d1
-    add             v1.4h, v7.4h, v5.4h
-    sub             v3.4h, v7.4h, v5.4h
-
-    trn1            v4.4h, v0.4h, v1.4h
-    trn2            v5.4h, v0.4h, v1.4h
-
-    trn1            v6.4h, v2.4h, v3.4h
-    trn2            v7.4h, v2.4h, v3.4h
-
-    add             v0.4h, v4.4h, v5.4h
-    sub             v1.4h, v4.4h, v5.4h
-
-    add             v2.4h, v6.4h, v7.4h
-    sub             v3.4h, v6.4h, v7.4h
-
-    trn1            v4.2s, v0.2s, v1.2s
-    trn2            v5.2s, v0.2s, v1.2s
-
-    trn1            v6.2s, v2.2s, v3.2s
-    trn2            v7.2s, v2.2s, v3.2s
-
-    abs             v4.4h, v4.4h
-    abs             v5.4h, v5.4h
-    abs             v6.4h, v6.4h
-    abs             v7.4h, v7.4h
-
-    smax            v1.4h, v4.4h, v5.4h
-    smax            v2.4h, v6.4h, v7.4h
-
-    add             v0.4h, v1.4h, v2.4h
-    uaddlp          v0.2s, v0.4h
-    uaddlp          v0.1d, v0.2s
-.endm
-
-// int satd_4x4(const pixel* pix1, intptr_t stride_pix1, const pixel* pix2, intptr_t stride_pix2)
-function PFX(pixel_satd_4x4_neon)
-    satd_4x4_neon
-    fmov            x0, d0
-    ret
-endfunc
-
-.macro x265_satd_4x8_8x4_end_neon
-    add             v0.8h, v4.8h, v6.8h
-    add             v1.8h, v5.8h, v7.8h
-    sub             v2.8h, v4.8h, v6.8h
-    sub             v3.8h, v5.8h, v7.8h
-
-    trn1            v16.8h, v0.8h, v1.8h
-    trn2            v17.8h, v0.8h, v1.8h
-    add             v4.8h, v16.8h, v17.8h
-    trn1            v18.8h, v2.8h, v3.8h
-    trn2            v19.8h, v2.8h, v3.8h
-    sub             v5.8h, v16.8h, v17.8h
-    add             v6.8h, v18.8h, v19.8h
-    sub             v7.8h, v18.8h, v19.8h
-    trn1            v0.4s, v4.4s, v6.4s
-    trn2            v2.4s, v4.4s, v6.4s
-    abs             v0.8h, v0.8h
-    trn1            v1.4s, v5.4s, v7.4s
-    trn2            v3.4s, v5.4s, v7.4s
-    abs             v2.8h, v2.8h
-    abs             v1.8h, v1.8h
-    abs             v3.8h, v3.8h
-    umax            v0.8h, v0.8h, v2.8h
-    umax            v1.8h, v1.8h, v3.8h
-    add             v0.8h, v0.8h, v1.8h
-    uaddlv          s0, v0.8h
-.endm
-
-.macro pixel_satd_4x8_neon
-    ld1r            {v1.2s}, x2, x3
-    ld1r            {v0.2s}, x0, x1
-    ld1r            {v3.2s}, x2, x3
-    ld1r            {v2.2s}, x0, x1
-    ld1r            {v5.2s}, x2, x3
-    ld1r            {v4.2s}, x0, x1
-    ld1r            {v7.2s}, x2, x3
-    ld1r            {v6.2s}, x0, x1
-
-    ld1             {v1.s}1, x2, x3
-    ld1             {v0.s}1, x0, x1
-    usubl           v0.8h, v0.8b, v1.8b
-    ld1             {v3.s}1, x2, x3
-    ld1             {v2.s}1, x0, x1
-    usubl           v1.8h, v2.8b, v3.8b
-    ld1             {v5.s}1, x2, x3
-    ld1             {v4.s}1, x0, x1
-    usubl           v2.8h, v4.8b, v5.8b
-    ld1             {v7.s}1, x2, x3
-    add             v4.8h, v0.8h, v1.8h
-    sub             v5.8h, v0.8h, v1.8h
-    ld1             {v6.s}1, x0, x1
-    usubl           v3.8h, v6.8b, v7.8b
-    add             v6.8h, v2.8h, v3.8h
-    sub             v7.8h, v2.8h, v3.8h
-    x265_satd_4x8_8x4_end_neon
-.endm
-
-// template<int w, int h>
-// int satd4(const pixel* pix1, intptr_t stride_pix1, const pixel* pix2, intptr_t stride_pix2)
-function PFX(pixel_satd_4x8_neon)
-    pixel_satd_4x8_neon
-    mov             w0, v0.s0
-    ret
-endfunc
-
-function PFX(pixel_satd_4x16_neon)
-    mov             w4, #0
-    pixel_satd_4x8_neon
-    mov             w5, v0.s0
-    add             w4, w4, w5
-    pixel_satd_4x8_neon
-    mov             w5, v0.s0
-    add             w0, w5, w4
-    ret
-endfunc
-
-function PFX(pixel_satd_4x32_neon)
-    mov             w4, #0
-.rept 4
-    pixel_satd_4x8_neon
-    mov             w5, v0.s0
-    add             w4, w4, w5
-.endr
-    mov             w0, w4
-    ret
-endfunc
-
-function PFX(pixel_satd_12x16_neon)
-    mov             x4, x0
-    mov             x5, x2
-    mov             w7, #0
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-
-    add             x0, x4, #4
-    add             x2, x5, #4
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-
-    add             x0, x4, #8
-    add             x2, x5, #8
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w0, w7, w6
-    ret
-endfunc
-
-function PFX(pixel_satd_12x32_neon)
-    mov             x4, x0
-    mov             x5, x2
-    mov             w7, #0
-.rept 4
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-.endr
-
-    add             x0, x4, #4
-    add             x2, x5, #4
-.rept 4
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-.endr
-
-    add             x0, x4, #8
-    add             x2, x5, #8
-.rept 4
-    pixel_satd_4x8_neon
-    mov             w6, v0.s0
-    add             w7, w7, w6
-.endr
-
-    mov             w0, w7
-    ret
-endfunc
-
-function PFX(pixel_satd_8x4_neon)
-    mov             x4, x0
-    mov             x5, x2
-    satd_4x4_neon
-    add             x0, x4, #4
-    add             x2, x5, #4
-    umov            x6, v0.d0
-    satd_4x4_neon
-    umov            x0, v0.d0
-    add             x0, x0, x6
-    ret
-endfunc
-
-.macro LOAD_DIFF_8x4 v0 v1 v2 v3
-    ld1             {v0.8b}, x0, x1
-    ld1             {v1.8b}, x2, x3
-    ld1             {v2.8b}, x0, x1
-    ld1             {v3.8b}, x2, x3
-    ld1             {v4.8b}, x0, x1
-    ld1             {v5.8b}, x2, x3
-    ld1             {v6.8b}, x0, x1
-    ld1             {v7.8b}, x2, x3
-    usubl           \v0, v0.8b, v1.8b
-    usubl           \v1, v2.8b, v3.8b
-    usubl           \v2, v4.8b, v5.8b
-    usubl           \v3, v6.8b, v7.8b
-.endm
-
-.macro LOAD_DIFF_16x4 v0 v1 v2 v3 v4 v5 v6 v7
-    ld1             {v0.16b}, x0, x1
-    ld1             {v1.16b}, x2, x3
-    ld1             {v2.16b}, x0, x1
-    ld1             {v3.16b}, x2, x3
-    ld1             {v4.16b}, x0, x1
-    ld1             {v5.16b}, x2, x3
-    ld1             {v6.16b}, x0, x1
-    ld1             {v7.16b}, x2, x3
-    usubl           \v0, v0.8b, v1.8b
-    usubl           \v1, v2.8b, v3.8b
-    usubl           \v2, v4.8b, v5.8b
-    usubl           \v3, v6.8b, v7.8b
-    usubl2          \v4, v0.16b, v1.16b
-    usubl2          \v5, v2.16b, v3.16b
-    usubl2          \v6, v4.16b, v5.16b
-    usubl2          \v7, v6.16b, v7.16b
-.endm
-
-function PFX(satd_16x4_neon), export=0
-    LOAD_DIFF_16x4  v16.8h, v17.8h, v18.8h, v19.8h, v20.8h, v21.8h, v22.8h, v23.8h
-    b               PFX(satd_8x4v_8x8h_neon)
-endfunc
-
-function PFX(satd_8x8_neon), export=0
-    LOAD_DIFF_8x4   v16.8h, v17.8h, v18.8h, v19.8h
-    LOAD_DIFF_8x4   v20.8h, v21.8h, v22.8h, v23.8h
-    b               PFX(satd_8x4v_8x8h_neon)
-endfunc
-
-// one vertical hadamard pass and two horizontal
-function PFX(satd_8x4v_8x8h_neon), export=0
-    HADAMARD4_V     v16.8h, v18.8h, v17.8h, v19.8h, v0.8h, v2.8h, v1.8h, v3.8h
-    HADAMARD4_V     v20.8h, v21.8h, v22.8h, v23.8h, v0.8h, v1.8h, v2.8h, v3.8h
-    trn4            v0.8h, v1.8h, v2.8h, v3.8h, v16.8h, v17.8h, v18.8h, v19.8h
-    trn4            v4.8h, v5.8h, v6.8h, v7.8h, v20.8h, v21.8h, v22.8h, v23.8h
-    SUMSUB_ABCD     v16.8h, v17.8h, v18.8h, v19.8h, v0.8h, v1.8h, v2.8h, v3.8h
-    SUMSUB_ABCD     v20.8h, v21.8h, v22.8h, v23.8h, v4.8h, v5.8h, v6.8h, v7.8h
-    trn4            v0.4s, v2.4s, v1.4s, v3.4s, v16.4s, v18.4s, v17.4s, v19.4s
-    trn4            v4.4s, v6.4s, v5.4s, v7.4s, v20.4s, v22.4s, v21.4s, v23.4s
-    ABS8            v0.8h, v1.8h, v2.8h, v3.8h, v4.8h, v5.8h, v6.8h, v7.8h
-    smax            v0.8h, v0.8h, v2.8h
-    smax            v1.8h, v1.8h, v3.8h
-    smax            v2.8h, v4.8h, v6.8h
-    smax            v3.8h, v5.8h, v7.8h
-    ret
-endfunc
-
-function PFX(pixel_satd_8x8_neon)
-    mov             x10, x30
-    bl              PFX(satd_8x8_neon)
-    add             v0.8h, v0.8h, v1.8h
-    add             v1.8h, v2.8h, v3.8h
-    add             v0.8h, v0.8h, v1.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_8x12_neon)
-    mov             x4, x0
-    mov             x5, x2
-    mov             x7, #0
-    satd_4x4_neon
-    umov            x6, v0.d0
-    add             x7, x7, x6
-    add             x0, x4, #4
-    add             x2, x5, #4
-    satd_4x4_neon
-    umov            x6, v0.d0
-    add             x7, x7, x6
-.rept 2
-    sub             x0, x0, #4
-    sub             x2, x2, #4
-    mov             x4, x0
-    mov             x5, x2
-    satd_4x4_neon
-    umov            x6, v0.d0
-    add             x7, x7, x6
-    add             x0, x4, #4
-    add             x2, x5, #4
-    satd_4x4_neon
-    umov            x6, v0.d0
-    add             x7, x7, x6
-.endr
-    mov             x0, x7
-    ret
-endfunc
-
-function PFX(pixel_satd_8x16_neon)
-    mov             x10, x30
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_8x32_neon)
-    mov             x10, x30
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 3
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_8x64_neon)
-    mov             x10, x30
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 7
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x4_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x8_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x12_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 2
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x16_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 3
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x24_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 5
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-.macro pixel_satd_16x32_neon
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 7
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-.endm
-
-function PFX(pixel_satd_16x32_neon)
-    mov             x10, x30
-    pixel_satd_16x32_neon
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_16x64_neon)
-    mov             x10, x30
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v0.8h, v1.8h
-    add             v31.8h, v2.8h, v3.8h
-.rept 15
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_24x32_neon)
-    mov             x10, x30
-    mov             x7, #0
-    mov             x4, x0
-    mov             x5, x2
-.rept 3
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-.rept 4
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-    add             x7, x7, x6
-    add             x4, x4, #8
-    add             x5, x5, #8
-    mov             x0, x4
-    mov             x2, x5
-.endr
-    mov             x0, x7
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_24x64_neon)
-    mov             x10, x30
-    mov             x7, #0
-    mov             x4, x0
-    mov             x5, x2
-.rept 3
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-.rept 4
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-    add             x7, x7, x6
-    add             x4, x4, #8
-    add             x5, x5, #8
-    mov             x0, x4
-    mov             x2, x5
-.endr
-    sub             x4, x4, #24
-    sub             x5, x5, #24
-    add             x0, x4, x1, lsl #5
-    add             x2, x5, x3, lsl #5
-    mov             x4, x0
-    mov             x5, x2
-.rept 3
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-.rept 4
-    bl              PFX(satd_8x8_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-    add             x7, x7, x6
-    add             x4, x4, #8
-    add             x5, x5, #8
-    mov             x0, x4
-    mov             x2, x5
-.endr
-    mov             x0, x7
-    ret             x10
-endfunc
-
-.macro pixel_satd_32x8
-    mov             x4, x0
-    mov             x5, x2
-.rept 2
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-    add             x0, x4, #16
-    add             x2, x5, #16
-.rept 2
-    bl              PFX(satd_16x4_neon)
-    add             v30.8h, v30.8h, v0.8h
-    add             v31.8h, v31.8h, v1.8h
-    add             v30.8h, v30.8h, v2.8h
-    add             v31.8h, v31.8h, v3.8h
-.endr
-.endm
-
-.macro satd_32x16_neon
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-    pixel_satd_32x8
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    pixel_satd_32x8
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-.endm
-
-.macro satd_64x16_neon
-    mov             x8, x0
-    mov             x9, x2
-    satd_32x16_neon
-    add             x7, x7, x6
-    add             x0, x8, #32
-    add             x2, x9, #32
-    satd_32x16_neon
-    add             x7, x7, x6
-.endm
-
-function PFX(pixel_satd_32x8_neon)
-    mov             x10, x30
-    mov             x7, #0
-    mov             x4, x0
-    mov             x5, x2
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-    pixel_satd_32x8
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x16_neon)
-    mov             x10, x30
-    satd_32x16_neon
-    mov             x0, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x24_neon)
-    mov             x10, x30
-    satd_32x16_neon
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    pixel_satd_32x8
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    add             x0, x0, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x32_neon)
-    mov             x10, x30
-    mov             x7, #0
-    satd_32x16_neon
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    add             x7, x7, x6
-    satd_32x16_neon
-    add             x0, x7, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x48_neon)
-    mov             x10, x30
-    mov             x7, #0
-.rept 2
-    satd_32x16_neon
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    add             x7, x7, x6
-.endr
-    satd_32x16_neon
-    add             x0, x7, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_32x64_neon)
-    mov             x10, x30
-    mov             x7, #0
-.rept 3
-    satd_32x16_neon
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    add             x7, x7, x6
-.endr
-    satd_32x16_neon
-    add             x0, x7, x6
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_64x16_neon)
-    mov             x10, x30
-    mov             x7, #0
-    satd_64x16_neon
-    mov             x0, x7
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_64x32_neon)
-    mov             x10, x30
-    mov             x7, #0
-    satd_64x16_neon
-    sub             x0, x0, #48
-    sub             x2, x2, #48
-    satd_64x16_neon
-    mov             x0, x7
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_64x48_neon)
-    mov             x10, x30
-    mov             x7, #0
-.rept 2
-    satd_64x16_neon
-    sub             x0, x0, #48
-    sub             x2, x2, #48
-.endr
-    satd_64x16_neon
-    mov             x0, x7
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_64x64_neon)
-    mov             x10, x30
-    mov             x7, #0
-.rept 3
-    satd_64x16_neon
-    sub             x0, x0, #48
-    sub             x2, x2, #48
-.endr
-    satd_64x16_neon
-    mov             x0, x7
-    ret             x10
-endfunc
-
-function PFX(pixel_satd_48x64_neon)
-    mov             x10, x30
-    mov             x7, #0
-    mov             x8, x0
-    mov             x9, x2
-.rept 3
-    satd_32x16_neon
-    sub             x0, x0, #16
-    sub             x2, x2, #16
-    add             x7, x7, x6
-.endr
-    satd_32x16_neon
-    add             x7, x7, x6
-
-    add             x0, x8, #32
-    add             x2, x9, #32
-    pixel_satd_16x32_neon
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-    add             x7, x7, x6
-
-    movi            v30.8h, #0
-    movi            v31.8h, #0
-    pixel_satd_16x32_neon
-    add             v0.8h, v30.8h, v31.8h
-    uaddlv          s0, v0.8h
-    mov             w6, v0.s0
-    add             x0, x7, x6
-    ret             x10
-endfunc
-
-function PFX(sa8d_8x8_neon), export=0
-    LOAD_DIFF_8x4   v16.8h, v17.8h, v18.8h, v19.8h
-    LOAD_DIFF_8x4   v20.8h, v21.8h, v22.8h, v23.8h
-    HADAMARD4_V     v16.8h, v18.8h, v17.8h, v19.8h, v0.8h, v2.8h, v1.8h, v3.8h
-    HADAMARD4_V     v20.8h, v21.8h, v22.8h, v23.8h, v0.8h, v1.8h, v2.8h, v3.8h
-    SUMSUB_ABCD     v0.8h, v16.8h, v1.8h, v17.8h, v16.8h, v20.8h, v17.8h, v21.8h
-    SUMSUB_ABCD     v2.8h, v18.8h, v3.8h, v19.8h, v18.8h, v22.8h, v19.8h, v23.8h
-    trn4            v4.8h, v5.8h, v6.8h, v7.8h, v0.8h, v1.8h, v2.8h, v3.8h
-    trn4            v20.8h, v21.8h, v22.8h, v23.8h, v16.8h, v17.8h, v18.8h, v19.8h
-    SUMSUB_ABCD     v2.8h, v3.8h, v24.8h, v25.8h, v20.8h, v21.8h, v4.8h, v5.8h
-    SUMSUB_ABCD     v0.8h, v1.8h, v4.8h, v5.8h, v22.8h, v23.8h, v6.8h, v7.8h
-    trn4            v20.4s, v22.4s, v21.4s, v23.4s, v2.4s, v0.4s, v3.4s, v1.4s
-    trn4            v16.4s, v18.4s, v17.4s, v19.4s, v24.4s, v4.4s, v25.4s, v5.4s
-    SUMSUB_ABCD     v0.8h, v2.8h, v1.8h, v3.8h, v20.8h, v22.8h, v21.8h, v23.8h
-    SUMSUB_ABCD     v4.8h, v6.8h, v5.8h, v7.8h, v16.8h, v18.8h, v17.8h, v19.8h
-    trn4            v16.2d, v20.2d, v17.2d, v21.2d, v0.2d, v4.2d, v1.2d, v5.2d
-    trn4            v18.2d, v22.2d, v19.2d, v23.2d, v2.2d, v6.2d, v3.2d, v7.2d
-    ABS8            v16.8h, v17.8h, v18.8h, v19.8h, v20.8h, v21.8h, v22.8h, v23.8h
-    smax            v16.8h, v16.8h, v20.8h
-    smax            v17.8h, v17.8h, v21.8h
-    smax            v18.8h, v18.8h, v22.8h
-    smax            v19.8h, v19.8h, v23.8h
-    add             v0.8h, v16.8h, v17.8h
-    add             v1.8h, v18.8h, v19.8h
-    ret
-endfunc
-
-function PFX(pixel_sa8d_8x8_neon)
-    mov             x10, x30
-    bl              PFX(sa8d_8x8_neon)
-    add             v0.8h, v0.8h, v1.8h
-    uaddlv          s0, v0.8h
-    mov             w0, v0.s0
-    add             w0, w0, #1
-    lsr             w0, w0, #1
-    ret             x10
-endfunc
-
-function PFX(pixel_sa8d_8x16_neon)
-    mov             x10, x30
-    bl              PFX(sa8d_8x8_neon)
-    add             v0.8h, v0.8h, v1.8h
-    uaddlv          s0, v0.8h
-    mov             w5, v0.s0
-    add             w5, w5, #1
-    lsr             w5, w5, #1
-    bl              PFX(sa8d_8x8_neon)
-    add             v0.8h, v0.8h, v1.8h
-    uaddlv          s0, v0.8h
-    mov             w4, v0.s0
-    add             w4, w4, #1
-    lsr             w4, w4, #1
-    add             w0, w4, w5
-    ret             x10
-endfunc
-
-.macro sa8d_16x16 reg
-    bl              PFX(sa8d_8x8_neon)
-    uaddlp          v30.4s, v0.8h
-    uaddlp          v31.4s, v1.8h
-    bl              PFX(sa8d_8x8_neon)
-    uadalp          v30.4s, v0.8h
-    uadalp          v31.4s, v1.8h
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    bl              PFX(sa8d_8x8_neon)
-    uadalp          v30.4s, v0.8h
-    uadalp          v31.4s, v1.8h
-    bl              PFX(sa8d_8x8_neon)
-    uadalp          v30.4s, v0.8h
-    uadalp          v31.4s, v1.8h
-    add             v0.4s, v30.4s, v31.4s
-    addv            s0, v0.4s
-    mov             \reg, v0.s0
-    add             \reg, \reg, #1
-    lsr             \reg, \reg, #1
-.endm
-
-function PFX(pixel_sa8d_16x16_neon)
-    mov             x10, x30
-    sa8d_16x16      w0
-    ret             x10
-endfunc
-
-function PFX(pixel_sa8d_16x32_neon)
-    mov             x10, x30
-    sa8d_16x16      w4
-    sub             x0, x0, #8
-    sub             x2, x2, #8
-    sa8d_16x16      w5
-    add             w0, w4, w5
-    ret             x10
-endfunc
-
-function PFX(pixel_sa8d_32x32_neon)
-    mov             x10, x30
-    sa8d_16x16      w4
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w5
-    sub             x0, x0, #24
-    sub             x2, x2, #24
-    sa8d_16x16      w6
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w7
-    add             w4, w4, w5
-    add             w6, w6, w7
-    add             w0, w4, w6
-    ret             x10
-endfunc
-
-function PFX(pixel_sa8d_32x64_neon)
-    mov             x10, x30
-    mov             w11, #4
-    mov             w9, #0
-.Loop_sa8d_32:
-    sub             w11, w11, #1
-    sa8d_16x16      w4
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w5
-    add             w4, w4, w5
-    add             w9, w9, w4
-    sub             x0, x0, #24
-    sub             x2, x2, #24
-    cbnz            w11, .Loop_sa8d_32
-    mov             w0, w9
-    ret             x10
-endfunc
-
-function PFX(pixel_sa8d_64x64_neon)
-    mov             x10, x30
-    mov             w11, #4
-    mov             w9, #0
-.Loop_sa8d_64:
-    sub             w11, w11, #1
-    sa8d_16x16      w4
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w5
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w6
-    sub             x0, x0, x1, lsl #4
-    sub             x2, x2, x3, lsl #4
-    add             x0, x0, #8
-    add             x2, x2, #8
-    sa8d_16x16      w7
-    add             w4, w4, w5
-    add             w6, w6, w7
-    add             w8, w4, w6
-    add             w9, w9, w8
-
-    sub             x0, x0, #56
-    sub             x2, x2, #56
-    cbnz            w11, .Loop_sa8d_64
-    mov             w0, w9
-    ret             x10
-endfunc
-
 /***** dequant_scaling*****/
 // void dequant_scaling_c(const int16_t* quantCoef, const int32_t* deQuantCoef, int16_t* coef, int num, int per, int shift)
 function PFX(dequant_scaling_neon)
@@ -1621,7 +383,16 @@
 
 // void dequant_normal_c(const int16_t* quantCoef, int16_t* coef, int num, int scale, int shift)
 function PFX(dequant_normal_neon)
-    lsr             w2, w2, #4              // num / 16
+//  X265_CHECK(num >= 4 * 4, "dequant num %d too small\n", num);
+//  X265_CHECK(num <= 32 * 32, "dequant num %d too large\n", num);
+//  X265_CHECK((num % 16) == 0, "dequant num %d not multiple of 16\n", num);
+#if HIGH_BIT_DEPTH
+    cmp             w3, #32768
+    blt             .dqn_skip
+    lsr             w3, w3, #(BIT_DEPTH - 8)
+    sub             w4, w4, #(BIT_DEPTH - 8)
+.dqn_skip:
+#endif
     neg             w4, w4
     dup             v0.8h, w3
     dup             v1.4s, w4
@@ -1643,7 +414,7 @@
     sqxtn           v3.4h, v18.4s
     sqxtn2          v3.8h, v19.4s
 
-    sub             w2, w2, #1
+    sub             w2, w2, #16
     st1             {v2.8h, v3.8h}, x1, #32
     cbnz            w2, .dqn_loop1
     ret
@@ -1709,88 +480,6 @@
     ret
 endfunc
 
-// int psyCost_pp(const pixel* source, intptr_t sstride, const pixel* recon, intptr_t rstride)
-function PFX(psyCost_4x4_neon)
-    ld1r            {v4.2s}, x0, x1
-    ld1r            {v5.2s}, x0, x1
-    ld1             {v4.s}1, x0, x1
-    ld1             {v5.s}1, x0, x1
-
-    ld1r            {v6.2s}, x2, x3
-    ld1r            {v7.2s}, x2, x3
-    ld1             {v6.s}1, x2, x3
-    ld1             {v7.s}1, x2, x3
-
-    uaddl           v2.8h, v4.8b, v5.8b
-    usubl           v3.8h, v4.8b, v5.8b
-    uaddl           v18.8h, v6.8b, v7.8b
-    usubl           v19.8h, v6.8b, v7.8b
-
-    mov             v20.d0, v2.d1
-    add             v0.4h, v2.4h, v20.4h
-    sub             v1.4h, v2.4h, v20.4h
-    mov             v21.d0, v3.d1
-    add             v22.4h, v3.4h, v21.4h
-    sub             v23.4h, v3.4h, v21.4h
-
-    mov             v24.d0, v18.d1
-    add             v16.4h, v18.4h, v24.4h
-    sub             v17.4h, v18.4h, v24.4h
-    mov             v25.d0, v19.d1
-    add             v26.4h, v19.4h, v25.4h
-    sub             v27.4h, v19.4h, v25.4h
-
-    mov             v0.d1, v22.d0
-    mov             v1.d1, v23.d0
-    trn1            v22.8h, v0.8h, v1.8h
-    trn2            v23.8h, v0.8h, v1.8h
-    mov             v16.d1, v26.d0
-    mov             v17.d1, v27.d0
-    trn1            v26.8h, v16.8h, v17.8h
-    trn2            v27.8h, v16.8h, v17.8h
-
-    add             v2.8h, v22.8h, v23.8h
-    sub             v3.8h, v22.8h, v23.8h
-    add             v18.8h, v26.8h, v27.8h
-    sub             v19.8h, v26.8h, v27.8h
-
-    uaddl           v20.8h, v4.8b, v5.8b
-    uaddl           v21.8h, v6.8b, v7.8b
-
-    trn1            v0.4s, v2.4s, v3.4s
-    trn2            v1.4s, v2.4s, v3.4s
-    trn1            v16.4s, v18.4s, v19.4s
-    trn2            v17.4s, v18.4s, v19.4s
-    abs             v0.8h, v0.8h
-    abs             v16.8h, v16.8h
-    abs             v1.8h, v1.8h
-    abs             v17.8h, v17.8h
-
-    uaddlv          s20, v20.8h
-    uaddlv          s21, v21.8h
-    mov             v20.s1, v21.s0
-
-    smax            v0.8h, v0.8h, v1.8h
-    smax            v16.8h, v16.8h, v17.8h
-
-    trn1            v4.2d, v0.2d, v16.2d
-    trn2            v5.2d, v0.2d, v16.2d
-    add             v0.8h, v4.8h, v5.8h
-    mov             v4.d0, v0.d1
-    uaddlv          s0, v0.4h
-    uaddlv          s4, v4.4h
-
-    ushr            v20.2s, v20.2s, #2
-    mov             v0.s1, v4.s0
-    sub             v0.2s, v0.2s, v20.2s
-    mov             w0, v0.s0
-    mov             w1, v0.s1
-    subs            w0, w0, w1
-    cneg            w0, w0, mi
-
-    ret
-endfunc
-
 // uint32_t quant_c(const int16_t* coef, const int32_t* quantCoeff, int32_t* deltaU, int16_t* qCoef, int qBits, int add, int numCoeff)
 function PFX(quant_neon)
     mov             w9, #1
@@ -2096,108 +785,6 @@
     ret
 endfunc
 
-// void weight_pp_c(const pixel* src, pixel* dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset)
-function PFX(weight_pp_neon)
-    sub             x2, x2, x3
-    ldr             w9, sp              // offset
-    lsl             w5, w5, #6            // w0 << correction
-
-    // count trailing zeros in w5 and compare against shift right amount.
-    rbit            w10, w5
-    clz             w10, w10
-    cmp             w10, w7
-    b.lt            .unfoldedShift
-
-    // shift right only removes trailing zeros: hoist LSR out of the loop.
-    lsr             w10, w5, w7           // w0 << correction >> shift
-    dup             v25.16b, w10
-    lsr             w6, w6, w7            // round >> shift
-    add             w6, w6, w9            // round >> shift + offset
-    dup             v26.8h, w6
-
-    // Check arithmetic range.
-    mov             w11, #255
-    madd            w11, w11, w10, w6
-    add             w11, w11, w9
-    lsr             w11, w11, #16
-    cbnz            w11, .widenTo32Bit
-
-    // 16-bit arithmetic is enough.
-.LoopHpp:
-    mov             x12, x3
-.LoopWpp:
-    ldr             q0, x0, #16
-    sub             x12, x12, #16
-    umull           v1.8h, v0.8b, v25.8b  // val *= w0 << correction >> shift
-    umull2          v2.8h, v0.16b, v25.16b
-    add             v1.8h, v1.8h, v26.8h  // val += round >> shift + offset
-    add             v2.8h, v2.8h, v26.8h
-    sqxtun          v0.8b, v1.8h          // val = x265_clip(val)
-    sqxtun2         v0.16b, v2.8h
-    str             q0, x1, #16
-    cbnz            x12, .LoopWpp
-    add             x1, x1, x2
-    add             x0, x0, x2
-    sub             x4, x4, #1
-    cbnz            x4, .LoopHpp
-    ret
-
-    // 32-bit arithmetic is needed.
-.widenTo32Bit:
-.LoopHpp32:
-    mov             x12, x3
-.LoopWpp32:
-    ldr             d0, x0, #8
-    sub             x12, x12, #8
-    uxtl            v0.8h, v0.8b
-    umull           v1.4s, v0.4h, v25.4h  // val *= w0 << correction >> shift
-    umull2          v2.4s, v0.8h, v25.8h
-    add             v1.4s, v1.4s, v26.4s  // val += round >> shift + offset
-    add             v2.4s, v2.4s, v26.4s
-    sqxtn           v0.4h, v1.4s          // val = x265_clip(val)
-    sqxtn2          v0.8h, v2.4s
-    sqxtun          v0.8b, v0.8h
-    str             d0, x1, #8
-    cbnz            x12, .LoopWpp32
-    add             x1, x1, x2
-    add             x0, x0, x2
-    sub             x4, x4, #1
-    cbnz            x4, .LoopHpp32
-    ret
-
-    // The shift right cannot be moved out of the loop.
-.unfoldedShift:
-    dup             v25.8h, w5            // w0 << correction
-    dup             v26.4s, w6            // round
-    neg             w7, w7                // -shift
-    dup             v27.4s, w7
-    dup             v29.4s, w9            // offset
-.LoopHppUS:
-    mov             x12, x3
-.LoopWppUS:
-    ldr             d0, x0, #8
-    sub             x12, x12, #8
-    uxtl            v0.8h, v0.8b
-    umull           v1.4s, v0.4h, v25.4h  // val *= w0
-    umull2          v2.4s, v0.8h, v25.8h
-    add             v1.4s, v1.4s, v26.4s  // val += round
-    add             v2.4s, v2.4s, v26.4s
-    sshl            v1.4s, v1.4s, v27.4s  // val >>= shift
-    sshl            v2.4s, v2.4s, v27.4s
-    add             v1.4s, v1.4s, v29.4s  // val += offset
-    add             v2.4s, v2.4s, v29.4s
-    sqxtn           v0.4h, v1.4s          // val = x265_clip(val)
-    sqxtn2          v0.8h, v2.4s
-    sqxtun          v0.8b, v0.8h
-    str             d0, x1, #8
-    cbnz            x12, .LoopWppUS
-    add             x1, x1, x2
-    add             x0, x0, x2
-    sub             x4, x4, #1
-    cbnz            x4, .LoopHppUS
-    ret
-endfunc
-
 // int scanPosLast(
 //     const uint16_t *scan,      // x0
 //     const coeff_t *coeff,      // x1
@@ -2208,27 +795,25 @@
 //     const uint16_t* scanCG4x4, // x6
 //     const int trSize)          // x7
 function PFX(scanPosLast_neon)
-    // convert unit of Stride(trSize) to int16_t
+    // Convert unit of trSize stride from elements (int16) to bytes.
     add             x7, x7, x7
 
-    // load scan table and convert to Byte
+    // Load scan table and convert to bytes.
     ldp             q0, q1, x6
-    xtn             v0.8b, v0.8h
-    xtn2            v0.16b, v1.8h   // v0 - Zigzag scan table
+    uzp1            v0.16b, v0.16b, v1.16b  // v0 - Zigzag scan table
 
     movrel          x10, g_SPL_and_mask
-    ldr             q28, x10      // v28 = mask for pmovmskb
-    movi            v31.16b, #0     // v31 = {0, ..., 0}
-    add             x10, x7, x7     // 2*x7
-    add             x11, x10, x7    // 3*x7
-    add             x9, x4, #1      // CG count
+    ldr             q28, x10              // v28 = mask for pmovmskb
+    add             x10, x7, x7             // 2*x7
+    add             x11, x7, x7, lsl #1     // 3*x7
+    add             x9, x4, #1              // CG count
 
 .Loop_spl:
-    // position of current CG
+    // Position of current CG.
     ldrh            w6, x0, #32
     add             x6, x1, x6, lsl #1
 
-    // loading current CG
+    // Loading current CG and saturate to bytes.
     ldr             d2, x6
     ldr             d3, x6, x7
     ldr             d4, x6, x10
@@ -2238,69 +823,60 @@
     sqxtn           v2.8b, v2.8h
     sqxtn2          v2.16b, v4.8h
 
-    // Zigzag
+    // Apply zigzag.
     tbl             v3.16b, {v2.16b}, v0.16b
 
-    // get sign
-    cmhi            v5.16b, v3.16b, v31.16b   // v5 = non-zero
-    cmlt            v3.16b, v3.16b, #0        // v3 = negative
+    // Get zero/sign.
+    cmeq            v5.16b, v3.16b, #0   // v5 = zero
+    cmlt            v3.16b, v3.16b, #0   // v3 = negative
 
-    // val - w13 = pmovmskb(v3)
+    //  val: w13 = pmovmskb(v3)
+    // mask: w15 = pmovmskb(v4)
     and             v3.16b, v3.16b, v28.16b
-    mov             d4, v3.d1
-    addv            b23, v3.8b
-    addv            b24, v4.8b
-    mov             v23.b1, v24.b0
-    fmov            w13, s23
-
-    // mask - w15 = pmovmskb(v5)
-    and             v5.16b, v5.16b, v28.16b
-    mov             d6, v5.d1
-    addv            b25, v5.8b
-    addv            b26, v6.8b
-    mov             v25.b1, v26.b0
-    fmov            w15, s25
+    bic             v4.16b, v28.16b, v5.16b
+    addp            v3.16b, v3.16b, v4.16b
+    addp            v3.16b, v3.16b, v3.16b
+    addp            v3.16b, v3.16b, v3.16b
+    fmov            w13, s3
+    lsr             w15, w13, #16
+
+    // coeffNum = addv(v3 != 0) = 16 - addv(v5)
+    addv            b5, v5.16b
+    smov            w6, v5.b0
+    add             w6, w6, #16
+    sub             w5, w5, w6
+    strb            w6, x4, #1
 
     // coeffFlag = reverse_bit(w15) in 16-bit
-    rbit            w12, w15
-    lsr             w12, w12, #16
-    fmov            s30, w12
+    rbit            w12, w13
     strh            w12, x3, #2
 
-    // accelerate by preparing w13 = w13 & w15
+    // Pack bits from w13 into w14, based on w15 mask.
     and             w13, w13, w15
     mov             x14, xzr
+    cbz             w15, .Lpext_end
 .Loop_spl_1:
-    cbz             w15, .pext_end
     clz             w6, w15
     lsl             w13, w13, w6
     lsl             w15, w15, w6
     extr            w14, w14, w13, #31
     bfm             w15, wzr, #1, #0
-    b               .Loop_spl_1
-.pext_end:
+    cbnz            w15, .Loop_spl_1
+.Lpext_end:
     strh            w14, x2, #2
 
-    // compute coeffNum = popcount(coeffFlag)
-    cnt             v30.8b, v30.8b
-    addp            v30.8b, v30.8b, v30.8b
-    fmov            w6, s30
-    sub             x5, x5, x6
-    strb            w6, x4, #1
-
     cbnz            x5, .Loop_spl
 
-    // count trailing zeros
+    // Count trailing zeros.
     rbit            w13, w12
     clz             w13, w13
     lsr             w12, w12, w13
     strh            w12, x3, #-2
 
-    // get last pos
+    // Get last pos.
     sub             x9, x4, x9
-    lsl             x0, x9, #4
     eor             w13, w13, #15
-    add             x0, x0, x13
+    add             x0, x13, x9, lsl #4
     ret
 endfunc

x265-4.1.tar/source/common/aarch64/sad-a.S -> x265-4.2.tar/source/common/aarch64/sad-a.S Changed

@@ -3,7 +3,8 @@
  *
  * Authors: Hongbin Liu <liuhongbin1@huawei.com>
  *          Sebastian Pop <spop@amazon.com>
-            Hari Limaye <hari.limaye@arm.com>
+ *          Hari Limaye <hari.limaye@arm.com>
+ *          Gerda Zsejke More <gerdazsejke.more@arm.com>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -35,6 +36,7 @@
 
 .text
 
+#if !HIGH_BIT_DEPTH
 .macro SAD_START_4 f
     ldr             s0, x0
     ldr             s1, x2
@@ -653,3 +655,814 @@
 const sad12_mask, align=8
 .byte 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0
 endconst
+
+#else // HIGH_BIT_DEPTH
+
+// int sad(const int16_t* pix1, intptr_t stride_pix1, const int16_t* pix2, intptr_t stride_pix2)
+.macro SAD_4 f
+    ld1             {v0.4h}, x0, x1
+    ld1             {v1.4h}, x2, x3
+    \f              v16.4s, v0.4h, v1.4h
+.endm
+
+.macro SAD_4xH h
+    SAD_4 uabdl
+.rept \h - 1
+    SAD_4 uabal
+.endr
+    addv            s0, v16.4s
+.endm
+
+.macro SAD_8x2 f
+    ld1             {v0.8h}, x0, x1
+    ld1             {v1.8h}, x2, x3
+    \f              v16.8h, v0.8h, v1.8h
+
+    ld1             {v0.8h}, x0, x1
+    ld1             {v1.8h}, x2, x3
+    \f              v17.8h, v0.8h, v1.8h
+.endm
+
+.macro SAD_8xH h
+    SAD_8x2 uabd
+.rept \h / 2 - 1
+    SAD_8x2 uaba
+.endr
+    uaddlp          v16.4s, v16.8h
+    uadalp          v16.4s, v17.8h
+    addv            s0, v16.4s
+.endm
+
+.macro SAD_FUNC w, h
+function PFX(pixel_sad_\w\()x\h\()_neon)
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    SAD_\w\()xH \h
+
+    fmov            w0, s0
+    ret
+endfunc
+.endm
+
+.macro SAD_12 f
+    ldr             q0, x0
+    ldr             q1, x2
+    ldr             d2, x0, #16
+    ldr             d3, x2, #16
+    \f              v16.8h, v0.8h, v1.8h
+    \f              v17.4h, v2.4h, v3.4h
+    add             x0, x0, x1
+    add             x2, x2, x3
+.endm
+
+.macro SAD_16 f
+    ld1             {v0.8h-v1.8h}, x0, x1
+    ld1             {v2.8h-v3.8h}, x2, x3
+    \f              v16.8h, v0.8h, v2.8h
+    \f              v17.8h, v1.8h, v3.8h
+.endm
+
+.macro SAD_32 f
+    ld1             {v0.8h-v3.8h}, x0, x1
+    ld1             {v4.8h-v7.8h}, x2, x3
+    \f              v16.8h, v0.8h, v4.8h
+    \f              v17.8h, v1.8h, v5.8h
+    \f              v18.8h, v2.8h, v6.8h
+    \f              v19.8h, v3.8h, v7.8h
+.endm
+
+.macro SAD_END_2_ACCUM
+    add             v16.8h, v16.8h, v17.8h
+    uaddlv          s0, v16.8h
+.endm
+
+.macro SAD_END_2_ACCUM_WIDEN
+    uaddlp          v16.4s, v16.8h
+    uadalp          v16.4s, v17.8h
+    addv            s0, v16.4s
+.endm
+
+.macro SAD_END_4_ACCUM_WIDEN
+    add             v16.8h, v16.8h, v17.8h
+    add             v18.8h, v18.8h, v19.8h
+    uaddlp          v16.4s, v16.8h
+    uadalp          v16.4s, v18.8h
+    addv            s0, v16.4s
+.endm
+
+.macro SAD_FUNC_LOOP w, h end_type
+function PFX(pixel_sad_\w\()x\h\()_neon)
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    SAD_\w uabd
+    SAD_\w uaba
+
+    mov             w9, #(\h - 2)/2
+
+.Loop_\w\()x\h:
+    sub             w9, w9, #1
+.rept 2
+    SAD_\w uaba
+.endr
+    cbnz            w9, .Loop_\w\()x\h
+
+    SAD_\end_type
+
+    fmov            w0, s0
+    ret
+endfunc
+.endm
+
+// SAD_<w>_WIDEN kernels widen into 32-bit accumulators.
+.macro SAD_16_WIDEN f
+    ld1             {v0.8h-v1.8h}, x0, x1
+    ld1             {v2.8h-v3.8h}, x2, x3
+    uabd            v18.8h, v0.8h, v2.8h
+    \f              v16.4s, v18.8h
+    uabd            v19.8h, v1.8h, v3.8h
+    \f              v17.4s, v19.8h
+.endm
+
+.macro SAD_24_WIDEN f
+    ld1             {v0.8h-v2.8h}, x0, x1
+    ld1             {v3.8h-v5.8h}, x2, x3
+    uabd            v19.8h, v0.8h, v3.8h
+    \f              v16.4s, v19.8h
+    uabd            v20.8h, v1.8h, v4.8h
+    \f              v17.4s, v20.8h
+    uabd            v21.8h, v2.8h, v5.8h
+    \f              v18.4s, v21.8h
+.endm
+
+.macro SAD_32_WIDEN f
+    ld1             {v0.8h-v3.8h}, x0, x1
+    ld1             {v4.8h-v7.8h}, x2, x3
+    uabd            v20.8h, v0.8h, v4.8h
+    \f              v16.4s, v20.8h
+    uabd            v21.8h, v1.8h, v5.8h
+    \f              v17.4s, v21.8h
+    uabd            v22.8h, v2.8h, v6.8h
+    \f              v18.4s, v22.8h
+    uabd            v23.8h, v3.8h, v7.8h
+    \f              v19.4s, v23.8h
+.endm
+
+.macro SAD_48_WIDEN f
+    ld1             {v0.8h-v3.8h}, x0
+    ld1             {v4.8h-v7.8h}, x2
+    uabd            v20.8h, v0.8h, v4.8h
+    \f              v16.4s, v20.8h
+    uabd            v21.8h, v1.8h, v5.8h
+    \f              v17.4s, v21.8h
+    uabd            v22.8h, v2.8h, v6.8h
+    \f              v18.4s, v22.8h
+    uabd            v23.8h, v3.8h, v7.8h
+    \f              v19.4s, v23.8h
+
+    ldp             q0, q1, x0, #64
+    ldp             q4, q5, x2, #64
+    uabd            v20.8h, v0.8h, v4.8h
+    uadalp          v16.4s, v20.8h
+    uabd            v21.8h, v1.8h, v5.8h
+    uadalp          v17.4s, v21.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+.endm
+
+.macro SAD_64_WIDEN f
+    ld1             {v0.8h-v3.8h}, x0
+    ld1             {v4.8h-v7.8h}, x2
+    uabd            v20.8h, v0.8h, v4.8h
+    \f              v16.4s, v20.8h
+    uabd            v21.8h, v1.8h, v5.8h
+    \f              v17.4s, v21.8h
+    uabd            v22.8h, v2.8h, v6.8h
+    \f              v18.4s, v22.8h
+    uabd            v23.8h, v3.8h, v7.8h
+    \f              v19.4s, v23.8h
+
+    ldp             q0, q1, x0, #64
+    ldp             q2, q3, x0, #96
+    ldp             q4, q5, x2, #64
+    ldp             q6, q7, x2, #96
+    uabd            v20.8h, v0.8h, v4.8h
+    uadalp          v16.4s, v20.8h
+    uabd            v21.8h, v1.8h, v5.8h
+    uadalp          v17.4s, v21.8h
+    uabd            v22.8h, v2.8h, v6.8h
+    uadalp          v18.4s, v22.8h
+    uabd            v23.8h, v3.8h, v7.8h
+    uadalp          v19.4s, v23.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+.endm
+
+
+.macro SAD_FUNC_LOOP_LARGE w, h
+function PFX(pixel_sad_\w\()x\h\()_neon)
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    SAD_\w\()_WIDEN uaddlp
+    SAD_\w\()_WIDEN uadalp
+
+    mov             w9, #(\h - 2)/2
+.Loop_\w\()x\h:
+    sub             w9, w9, #1
+.rept 2
+    SAD_\w\()_WIDEN uadalp
+.endr
+    cbnz            w9, .Loop_\w\()x\h
+
+    add             v16.4s, v16.4s, v17.4s
+.if \w != 16
+.if \w != 24
+    add             v18.4s, v18.4s, v19.4s
+.endif
+    add             v16.4s, v16.4s, v18.4s
+.endif
+    addv            s0, v16.4s
+
+    fmov            w0, s0
+    ret
+endfunc
+.endm
+
+SAD_FUNC  4, 4
+SAD_FUNC  4, 8
+SAD_FUNC  4, 16
+SAD_FUNC  8, 4
+SAD_FUNC  8, 8
+SAD_FUNC  8, 16
+SAD_FUNC  8, 32
+SAD_FUNC_LOOP  12, 16, END_2_ACCUM_WIDEN
+SAD_FUNC_LOOP  16, 4, END_2_ACCUM
+SAD_FUNC_LOOP  16, 8, END_2_ACCUM
+SAD_FUNC_LOOP  16, 12, END_2_ACCUM_WIDEN
+SAD_FUNC_LOOP  16, 16, END_2_ACCUM_WIDEN
+SAD_FUNC_LOOP  32, 8, END_4_ACCUM_WIDEN
+SAD_FUNC_LOOP_LARGE  16, 32
+SAD_FUNC_LOOP_LARGE  16, 64
+SAD_FUNC_LOOP_LARGE  24, 32
+SAD_FUNC_LOOP_LARGE  32, 16
+SAD_FUNC_LOOP_LARGE  32, 24
+SAD_FUNC_LOOP_LARGE  32, 32
+SAD_FUNC_LOOP_LARGE  32, 64
+SAD_FUNC_LOOP_LARGE  48, 64
+SAD_FUNC_LOOP_LARGE  64, 16
+SAD_FUNC_LOOP_LARGE  64, 32
+SAD_FUNC_LOOP_LARGE  64, 48
+SAD_FUNC_LOOP_LARGE  64, 64
+
+//void sad_x3(const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, intptr_t frefstride, int32_t* res)
+//void sad_x4(const pixel* pix1, const pixel* pix2, const pixel* pix3, const pixel* pix4, const pixel* pix5, intptr_t frefstride, int32_t* res)
+.macro SAD_xN_4 n, f
+    ld1             {v0.4h}, x0, x7
+    ld1             {v1.4h}, x1, x5
+    ld1             {v2.4h}, x2, x5
+    ld1             {v3.4h}, x3, x5
+    \f              v16.4s, v0.4h, v1.4h
+    \f              v17.4s, v0.4h, v2.4h
+    \f              v18.4s, v0.4h, v3.4h
+.if \n == 4
+    ld1             {v4.4h}, x4, x5
+    \f              v19.4s, v0.4h, v4.4h
+.endif
+.endm
+
+.macro SAD_xN_4xH n, h
+    SAD_xN_4 \n, uabdl
+.rept \h - 1
+    SAD_xN_4 \n, uabal
+.endr
+.endm
+
+.macro SAD_xN_8x2 n, f
+    ld1             {v0.8h}, x0, x7
+    ld1             {v1.8h}, x1, x5
+    ld1             {v2.8h}, x2, x5
+    ld1             {v3.8h}, x3, x5
+    \f              v16.8h, v0.8h, v1.8h
+    \f              v17.8h, v0.8h, v2.8h
+    \f              v18.8h, v0.8h, v3.8h
+.if \n == 4
+    ld1             {v4.8h}, x4, x5
+    \f              v22.8h, v0.8h, v4.8h
+.endif
+
+    ld1             {v0.8h}, x0, x7
+    ld1             {v1.8h}, x1, x5
+    ld1             {v2.8h}, x2, x5
+    ld1             {v3.8h}, x3, x5
+    \f              v19.8h, v0.8h, v1.8h
+    \f              v20.8h, v0.8h, v2.8h
+    \f              v21.8h, v0.8h, v3.8h
+.if \n == 4
+    ld1             {v4.8h}, x4, x5
+    \f              v23.8h, v0.8h, v4.8h
+.endif
+.endm
+
+.macro SAD_xN_8xH n, h
+    SAD_xN_8x2 \n, uabd
+.rept \h /2 - 1
+    SAD_xN_8x2 \n, uaba
+.endr
+    uaddlp          v16.4s, v16.8h
+    uadalp          v16.4s, v19.8h
+    uaddlp          v17.4s, v17.8h
+    uadalp          v17.4s, v20.8h
+    uaddlp          v18.4s, v18.8h
+    uadalp          v18.4s, v21.8h
+.if \n == 4
+    uaddlp          v19.4s, v22.8h
+    uadalp          v19.4s, v23.8h
+.endif
+.endm
+
+.macro SAD_xN_FUNC n, w, h
+function PFX(sad_x\n\()_\w\()x\h\()_neon)
+    // Make function arguments for n == 3 look like n == 4.
+.if \n == 3
+    mov             x6, x5
+    mov             x5, x4
+.endif
+
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x5, x5, x5
+    mov             x7, #(FENC_STRIDE << 1)
+
+    SAD_xN_\w\()xH \n, \h
+
+.if \n == 3
+    addp            v0.4s, v16.4s, v17.4s
+    addp            v1.4s, v18.4s, v18.4s
+    addp            v0.4s, v0.4s, v1.4s
+    str             d0, x6
+    add             x6, x6, #8
+    st1             {v0.s}2, x6
+.else
+    addp            v16.4s, v16.4s, v17.4s
+    addp            v18.4s, v18.4s, v19.4s
+    addp            v16.4s, v16.4s, v18.4s
+    str             q16, x6
+.endif
+
+    ret
+endfunc
+.endm
+
+.macro SAD_xN_12 n, f
+    ldr             q0, x0
+    ldr             q1, x1
+    ldr             q2, x2
+    ldr             q3, x3
+    ldr             d4, x0, #16
+    ldr             d5, x1, #16
+    ldr             d6, x2, #16
+    ldr             d7, x3, #16
+    \f              v16.8h, v0.8h, v1.8h
+    \f              v18.8h, v0.8h, v2.8h
+    \f              v20.8h, v0.8h, v3.8h
+    \f              v17.8h, v4.8h, v5.8h
+    \f              v19.8h, v4.8h, v6.8h
+    \f              v21.8h, v4.8h, v7.8h
+    add             x0, x0, x7
+    add             x1, x1, x5
+    add             x2, x2, x5
+    add             x3, x3, x5
+.if \n == 4
+    ldr             q3, x4
+    ldr             d7, x4, #16
+    \f              v22.8h, v0.8h, v3.8h
+    \f              v23.8h, v4.8h, v7.8h
+    add             x4, x4, x5
+.endif
+.endm
+
+.macro SAD_xN_16 n f
+    ld1             {v0.8h-v1.8h}, x0, x7
+    ld1             {v2.8h-v3.8h}, x1, x5
+    \f              v16.8h, v0.8h, v2.8h
+    \f              v17.8h, v1.8h, v3.8h
+    ld1             {v4.8h-v5.8h}, x2, x5
+    \f              v18.8h, v0.8h, v4.8h
+    \f              v19.8h, v1.8h, v5.8h
+    ld1             {v6.8h-v7.8h}, x3, x5
+    \f              v20.8h, v0.8h, v6.8h
+    \f              v21.8h, v1.8h, v7.8h
+.if \n == 4
+    ld1             {v6.8h-v7.8h}, x4, x5
+    \f              v22.8h, v0.8h, v6.8h
+    \f              v23.8h, v1.8h, v7.8h
+.endif
+.endm
+
+.macro SAD_xN_32 n f
+    ld1             {v0.8h-v3.8h}, x0, x7
+    ld1             {v4.8h-v7.8h}, x1, x5
+    \f              v16.8h, v0.8h, v4.8h
+    uaba            v16.8h, v1.8h, v5.8h
+    \f              v17.8h, v2.8h, v6.8h
+    uaba            v17.8h, v3.8h, v7.8h
+    ld1             {v4.8h-v7.8h}, x2, x5
+    \f              v18.8h, v0.8h, v4.8h
+    uaba            v18.8h, v1.8h, v5.8h
+    \f              v19.8h, v2.8h, v6.8h
+    uaba            v19.8h, v3.8h, v7.8h
+    ld1             {v4.8h-v7.8h}, x3, x5
+    \f              v20.8h, v0.8h, v4.8h
+    uaba            v20.8h, v1.8h, v5.8h
+    \f              v21.8h, v2.8h, v6.8h
+    uaba            v21.8h, v3.8h, v7.8h
+.if \n == 4
+    ld1             {v4.8h-v7.8h}, x4, x5
+    \f              v22.8h, v0.8h, v4.8h
+    uaba            v22.8h, v1.8h, v5.8h
+    \f              v23.8h, v2.8h, v6.8h
+    uaba            v23.8h, v3.8h, v7.8h
+.endif
+.endm
+
+.macro SAD_xN_FUNC_LOOP n, w, h end_type
+function PFX(sad_x\n\()_\w\()x\h\()_neon)
+    // Make function arguments for n == 3 look like n == 4.
+.if \n == 3
+    mov             x6, x5
+    mov             x5, x4
+.endif
+
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x5, x5, x5
+    mov             x7, #(FENC_STRIDE << 1)
+
+    SAD_xN_\w \n, uabd
+
+    mov             w8, #\h - 1
+.Loop_x\n\()_\w\()x\h:
+    sub             w8, w8, #1
+    SAD_xN_\w \n, uaba
+    cbnz            w8, .Loop_x\n\()_\w\()x\h
+
+    uaddlp          v16.4s, v16.8h
+    uadalp          v16.4s, v17.8h
+    uaddlp          v18.4s, v18.8h
+    uadalp          v18.4s, v19.8h
+    uaddlp          v20.4s, v20.8h
+    uadalp          v20.4s, v21.8h
+
+.if \n == 3
+    addp            v0.4s, v16.4s, v18.4s
+    addp            v1.4s, v20.4s, v20.4s
+    addp            v0.4s, v0.4s, v1.4s
+    str             d0, x6
+    add             x6, x6, #8
+    st1             {v0.s}2, x6
+.else
+    uaddlp          v22.4s, v22.8h
+    uadalp          v22.4s, v23.8h
+    addp            v16.4s, v16.4s, v18.4s
+    addp            v20.4s, v20.4s, v22.4s
+    addp            v16.4s, v16.4s, v20.4s
+    str             q16, x6
+.endif
+
+    ret
+endfunc
+.endm
+
+.macro SAD_xN_16_WIDEN n f
+    ld1             {v0.8h-v1.8h}, x0, x7
+    ld1             {v2.8h-v3.8h}, x1, x5
+    uabd            v22.8h, v0.8h, v2.8h
+    \f              v16.4s, v22.8h
+    uabd            v23.8h, v1.8h, v3.8h
+    \f              v17.4s, v23.8h
+    ld1             {v4.8h-v5.8h}, x2, x5
+    uabd            v24.8h, v0.8h, v4.8h
+    \f              v18.4s, v24.8h
+    uabd            v25.8h, v1.8h, v5.8h
+    \f              v19.4s, v25.8h
+    ld1             {v6.8h-v7.8h}, x3, x5
+    uabd            v26.8h, v0.8h, v6.8h
+    \f              v20.4s, v26.8h
+    uabd            v27.8h, v1.8h, v7.8h
+    \f              v21.4s, v27.8h
+.if \n == 4
+    ld1             {v2.8h-v3.8h}, x4, x5
+    uabd            v28.8h, v0.8h, v2.8h
+    \f              v30.4s, v28.8h
+    uabd            v29.8h, v1.8h, v3.8h
+    \f              v31.4s, v29.8h
+.endif
+.endm
+
+.macro SAD_xN_24_WIDEN n f
+    ld1             {v0.8h-v2.8h}, x0, x7
+    ld1             {v3.8h-v5.8h}, x1, x5
+    uabd            v6.8h, v0.8h, v3.8h
+    uaba            v6.8h, v1.8h, v4.8h
+    \f              v16.4s, v6.8h
+    uabd            v7.8h, v2.8h, v5.8h
+    \f              v17.4s, v7.8h
+    ld1             {v27.8h-v29.8h}, x2, x5
+    uabd            v22.8h, v0.8h, v27.8h
+    uaba            v22.8h, v1.8h, v28.8h
+    \f              v18.4s, v22.8h
+    uabd            v23.8h, v2.8h, v29.8h
+    \f              v19.4s, v23.8h
+    ld1             {v3.8h-v5.8h}, x3, x5
+    uabd            v24.8h, v0.8h, v3.8h
+    uaba            v24.8h, v1.8h, v4.8h
+    \f              v20.4s, v24.8h
+    uabd            v25.8h, v2.8h, v5.8h
+    \f              v21.4s, v25.8h
+.if \n == 4
+    ld1             {v27.8h-v29.8h}, x4, x5
+    uabd            v22.8h, v0.8h, v27.8h
+    uaba            v22.8h, v1.8h, v28.8h
+    \f              v30.4s, v22.8h
+    uabd            v23.8h, v2.8h, v29.8h
+    \f              v31.4s, v23.8h
+.endif
+.endm
+
+.macro SAD_xN_32_WIDEN n f
+    ld1             {v0.8h-v3.8h}, x0, x7
+    ld1             {v4.8h-v7.8h}, x1, x5
+    uabd            v22.8h, v0.8h, v4.8h
+    uaba            v22.8h, v1.8h, v5.8h
+    \f              v16.4s, v22.8h
+    uabd            v23.8h, v2.8h, v6.8h
+    uaba            v23.8h, v3.8h, v7.8h
+    \f              v17.4s, v23.8h
+
+    ld1             {v4.8h-v7.8h}, x2, x5
+    uabd            v24.8h, v0.8h, v4.8h
+    uaba            v24.8h, v1.8h, v5.8h
+    \f              v18.4s, v24.8h
+    uabd            v25.8h, v2.8h, v6.8h
+    uaba            v25.8h, v3.8h, v7.8h
+    \f              v19.4s, v25.8h
+
+    ld1             {v4.8h-v7.8h}, x3, x5
+    uabd            v26.8h, v0.8h, v4.8h
+    uaba            v26.8h, v1.8h, v5.8h
+    \f              v20.4s, v26.8h
+    uabd            v27.8h, v2.8h, v6.8h
+    uaba            v27.8h, v3.8h, v7.8h
+    \f              v21.4s, v27.8h
+
+.if \n == 4
+    ld1             {v4.8h-v7.8h}, x4, x5
+    uabd            v22.8h, v0.8h, v4.8h
+    uaba            v22.8h, v1.8h, v5.8h
+    \f              v30.4s, v22.8h
+    uabd            v23.8h, v2.8h, v6.8h
+    uaba            v23.8h, v3.8h, v7.8h
+    \f              v31.4s, v23.8h
+.endif
+.endm
+
+.macro SAD_xN_48_WIDEN n f
+    ld1             {v0.8h-v3.8h}, x0
+    ld1             {v26.8h-v29.8h}, x1
+    uabd            v6.8h, v0.8h, v26.8h
+    uaba            v6.8h, v1.8h, v27.8h
+    \f              v16.4s, v6.8h
+    uabd            v7.8h, v2.8h, v28.8h
+    uaba            v7.8h, v3.8h, v29.8h
+    \f              v17.4s, v7.8h
+    ldp             q4, q5, x0, #64
+    ldp             q26, q27, x1, #64
+    uabd            v22.8h, v4.8h, v26.8h
+    uaba            v22.8h, v5.8h, v27.8h
+    uadalp          v16.4s, v22.8h
+
+    ld1             {v26.8h-v29.8h}, x2
+    uabd            v23.8h, v0.8h, v26.8h
+    uaba            v23.8h, v1.8h, v27.8h
+    \f              v18.4s, v23.8h
+    uabd            v24.8h, v2.8h, v28.8h
+    uaba            v24.8h, v3.8h, v29.8h
+    \f              v19.4s, v24.8h
+    ldp             q26, q27, x2, #64
+    uabd            v25.8h, v4.8h, v26.8h
+    uaba            v25.8h, v5.8h, v27.8h
+    uadalp          v18.4s, v25.8h
+
+    ld1             {v26.8h-v29.8h}, x3
+    uabd            v6.8h, v0.8h, v26.8h
+    uaba            v6.8h, v1.8h, v27.8h
+    \f              v20.4s, v6.8h
+    uabd            v7.8h, v2.8h, v28.8h
+    uaba            v7.8h, v3.8h, v29.8h
+    \f              v21.4s, v7.8h
+    ldp             q26, q27, x3, #64
+    uabd            v22.8h, v4.8h, v26.8h
+    uaba            v22.8h, v5.8h, v27.8h
+    uadalp          v20.4s, v22.8h
+
+    add             x0, x0, x7
+    add             x1, x1, x5
+    add             x2, x2, x5
+    add             x3, x3, x5
+
+.if \n == 4
+    ld1             {v26.8h-v29.8h}, x4
+    uabd            v6.8h, v0.8h, v26.8h
+    uaba            v6.8h, v1.8h, v27.8h
+    \f              v30.4s, v6.8h
+    uabd            v7.8h, v2.8h, v28.8h
+    uaba            v7.8h, v3.8h, v29.8h
+    \f              v31.4s, v7.8h
+    ldp             q26, q27, x4, #64
+    uabd            v22.8h, v4.8h, v26.8h
+    uaba            v22.8h, v5.8h, v27.8h
+    uadalp          v30.4s, v22.8h
+    add             x4, x4, x5
+.endif
+.endm
+
+.macro SAD_xN_64_WIDEN n f
+    ld1             {v0.8h-v3.8h}, x0
+    ld1             {v26.8h-v29.8h}, x1
+    uabd            v22.8h, v0.8h, v26.8h
+    uaba            v22.8h, v1.8h, v27.8h
+    \f              v16.4s, v22.8h
+    uabd            v23.8h, v2.8h, v28.8h
+    uaba            v23.8h, v3.8h, v29.8h
+    \f              v17.4s, v23.8h
+    ldp             q4, q5, x0, #64
+    ldp             q6, q7, x0, #96
+    ldp             q26, q27, x1, #64
+    ldp             q28, q29, x1, #96
+    uabd            v24.8h, v4.8h, v26.8h
+    uaba            v24.8h, v5.8h, v27.8h
+    uadalp          v16.4s, v24.8h
+    uabd            v25.8h, v6.8h, v28.8h
+    uaba            v25.8h, v7.8h, v29.8h
+    uadalp          v17.4s, v25.8h
+
+    ld1             {v26.8h-v29.8h}, x2
+    uabd            v22.8h, v0.8h, v26.8h
+    uaba            v22.8h, v1.8h, v27.8h
+    \f              v18.4s, v22.8h
+    uabd            v23.8h, v2.8h, v28.8h
+    uaba            v23.8h, v3.8h, v29.8h
+    \f              v19.4s, v23.8h
+    ldp             q26, q27, x2, #64
+    ldp             q28, q29, x2, #96
+    uabd            v24.8h, v4.8h, v26.8h
+    uaba            v24.8h, v5.8h, v27.8h
+    uadalp          v18.4s, v24.8h
+    uabd            v25.8h, v6.8h, v28.8h
+    uaba            v25.8h, v7.8h, v29.8h
+    uadalp          v19.4s, v25.8h
+
+    ld1             {v26.8h-v29.8h}, x3
+    uabd            v22.8h, v0.8h, v26.8h
+    uaba            v22.8h, v1.8h, v27.8h
+    \f              v20.4s, v22.8h
+    uabd            v23.8h, v2.8h, v28.8h
+    uaba            v23.8h, v3.8h, v29.8h
+    \f              v21.4s, v23.8h
+    ldp             q26, q27, x3, #64
+    ldp             q28, q29, x3, #96
+    uabd            v24.8h, v4.8h, v26.8h
+    uaba            v24.8h, v5.8h, v27.8h
+    uadalp          v20.4s, v24.8h
+    uabd            v25.8h, v6.8h, v28.8h
+    uaba            v25.8h, v7.8h, v29.8h
+    uadalp          v21.4s, v25.8h
+
+    add             x0, x0, x7
+    add             x1, x1, x5
+    add             x2, x2, x5
+    add             x3, x3, x5
+
+.if \n == 4
+    ld1             {v26.8h-v29.8h}, x4
+    uabd            v22.8h, v0.8h, v26.8h
+    uaba            v22.8h, v1.8h, v27.8h
+    \f              v30.4s, v22.8h
+    uabd            v23.8h, v2.8h, v28.8h
+    uaba            v23.8h, v3.8h, v29.8h
+    \f              v31.4s, v23.8h
+    ldp             q26, q27, x4, #64
+    ldp             q28, q29, x4, #96
+    uabd            v24.8h, v4.8h, v26.8h
+    uaba            v24.8h, v5.8h, v27.8h
+    uadalp          v30.4s, v24.8h
+    uabd            v25.8h, v6.8h, v28.8h
+    uaba            v25.8h, v7.8h, v29.8h
+    uadalp          v31.4s, v25.8h
+    add             x4, x4, x5
+.endif
+.endm
+
+.macro SAD_xN_FUNC_LOOP_LARGE n, w, h
+function PFX(sad_x\n\()_\w\()x\h\()_neon)
+    // Make function arguments for n == 3 look like n == 4.
+.if \n == 3
+    mov             x6, x5
+    mov             x5, x4
+.endif
+
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add             x5, x5, x5
+    mov             x7, #(FENC_STRIDE << 1)
+
+    SAD_xN_\w\()_WIDEN \n, uaddlp
+    SAD_xN_\w\()_WIDEN \n, uadalp
+
+    mov             w8, #(\h - 2)/2
+.Loop_x\n\()_\w\()x\h:
+    sub             w8, w8, #1
+.rept 2
+    SAD_xN_\w\()_WIDEN \n, uadalp
+.endr
+    cbnz            w8, .Loop_x\n\()_\w\()x\h
+
+    add             v16.4s, v16.4s, v17.4s
+    add             v17.4s, v18.4s, v19.4s
+    add             v18.4s, v20.4s, v21.4s
+.if \n == 3
+    addp            v0.4s, v16.4s, v17.4s
+    addp            v1.4s, v18.4s, v18.4s
+    addp            v0.4s, v0.4s, v1.4s
+    str             d0, x6
+    add             x6, x6, #8
+    st1             {v0.s}2, x6
+.else
+    add             v19.4s, v30.4s, v31.4s
+    addp            v16.4s, v16.4s, v17.4s
+    addp            v18.4s, v18.4s, v19.4s
+    addp            v16.4s, v16.4s, v18.4s
+    str             q16, x6
+.endif
+
+    ret
+endfunc
+.endm
+
+SAD_xN_FUNC  3, 4, 4
+SAD_xN_FUNC  3, 4, 8
+SAD_xN_FUNC  3, 4, 16
+SAD_xN_FUNC  3, 8, 4
+SAD_xN_FUNC  3, 8, 8
+SAD_xN_FUNC  3, 8, 16
+SAD_xN_FUNC  3, 8, 32
+SAD_xN_FUNC_LOOP  3, 12, 16
+SAD_xN_FUNC_LOOP  3, 16, 4
+SAD_xN_FUNC_LOOP  3, 16, 8
+SAD_xN_FUNC_LOOP  3, 16, 12
+SAD_xN_FUNC_LOOP  3, 16, 16
+SAD_xN_FUNC_LOOP  3, 32, 8
+SAD_xN_FUNC_LOOP_LARGE  3, 16, 32
+SAD_xN_FUNC_LOOP_LARGE  3, 16, 64
+SAD_xN_FUNC_LOOP_LARGE  3, 24, 32
+SAD_xN_FUNC_LOOP_LARGE  3, 32, 16
+SAD_xN_FUNC_LOOP_LARGE  3, 32, 24
+SAD_xN_FUNC_LOOP_LARGE  3, 32, 32
+SAD_xN_FUNC_LOOP_LARGE  3, 32, 64
+SAD_xN_FUNC_LOOP_LARGE  3, 48, 64
+SAD_xN_FUNC_LOOP_LARGE  3, 64, 16
+SAD_xN_FUNC_LOOP_LARGE  3, 64, 32
+SAD_xN_FUNC_LOOP_LARGE  3, 64, 48
+SAD_xN_FUNC_LOOP_LARGE  3, 64, 64
+
+SAD_xN_FUNC  4, 4, 4
+SAD_xN_FUNC  4, 4, 8
+SAD_xN_FUNC  4, 4, 16
+SAD_xN_FUNC  4, 8, 4
+SAD_xN_FUNC  4, 8, 8
+SAD_xN_FUNC  4, 8, 16
+SAD_xN_FUNC  4, 8, 32
+SAD_xN_FUNC_LOOP  4, 12, 16
+SAD_xN_FUNC_LOOP  4, 16, 4
+SAD_xN_FUNC_LOOP  4, 16, 8
+SAD_xN_FUNC_LOOP  4, 16, 12
+SAD_xN_FUNC_LOOP  4, 16, 16
+SAD_xN_FUNC_LOOP  4, 32, 8
+SAD_xN_FUNC_LOOP_LARGE  4, 16, 32
+SAD_xN_FUNC_LOOP_LARGE  4, 16, 64
+SAD_xN_FUNC_LOOP_LARGE  4, 24, 32
+SAD_xN_FUNC_LOOP_LARGE  4, 32, 16
+SAD_xN_FUNC_LOOP_LARGE  4, 32, 24
+SAD_xN_FUNC_LOOP_LARGE  4, 32, 32
+SAD_xN_FUNC_LOOP_LARGE  4, 32, 64
+SAD_xN_FUNC_LOOP_LARGE  4, 48, 64
+SAD_xN_FUNC_LOOP_LARGE  4, 64, 16
+SAD_xN_FUNC_LOOP_LARGE  4, 64, 32
+SAD_xN_FUNC_LOOP_LARGE  4, 64, 48
+SAD_xN_FUNC_LOOP_LARGE  4, 64, 64
+
+#endif // !HIGH_BIT_DEPTH

x265-4.2.tar/source/common/aarch64/ssd-a-sve.S Added

@@ -0,0 +1,483 @@
+/*****************************************************************************
+ * Copyright (C) 2024 MulticoreWare, Inc
+ *
+ * Authors: Gerda Zsejke More <gerdazsejke.more@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+#include "ssd-a-common.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+
+.text
+
+.arch armv8-a+sve
+
+#if HIGH_BIT_DEPTH
+.macro SSE_PP_4x2
+    ldr             d16, x0
+    ldr             d17, x2
+    ldr             d18, x0, x1
+    ldr             d19, x2, x3
+    uabd            v2.4h, v16.4h, v17.4h
+    uabd            v3.4h, v18.4h, v19.4h
+    udot            z0.d, z2.h, z2.h
+    udot            z0.d, z3.h, z3.h
+.endm
+
+.macro SSE_PP_4xN h
+function PFX(pixel_sse_pp_4x\h\()_sve)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+.rept (\h / 2) - 1
+    SSE_PP_4x2
+    add             x0, x0, x1, lsl #1
+    add             x2, x2, x3, lsl #1
+.endr
+    SSE_PP_4x2
+
+    fmov            w0, s0
+    ret
+endfunc
+.endm
+
+SSE_PP_4xN 4
+SSE_PP_4xN 8
+
+.macro SSE_PP_8xN h
+function PFX(pixel_sse_pp_8x\h\()_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+.rept \h / 2
+    ld1             {v16.8h}, x0, x1
+    ld1             {v17.8h}, x2, x3
+    uabd            v2.8h, v16.8h, v17.8h
+    udot            z0.d, z2.h, z2.h
+    ld1             {v18.8h}, x0, x1
+    ld1             {v19.8h}, x2, x3
+    uabd            v3.8h, v18.8h, v19.8h
+    udot            z1.d, z3.h, z3.h
+.endr
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            w0, s0
+    ret
+endfunc
+.endm
+
+SSE_PP_8xN 8
+SSE_PP_8xN 16
+
+.macro SSE_PP_16xN h
+function PFX(pixel_sse_pp_16x\h\()_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, \h
+.Loop_sse_pp_16x\h:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0, x1
+    ld1             {v18.8h-v19.8h}, x2, x3
+    uabd            v2.8h, v16.8h, v18.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v17.8h, v19.8h
+    udot            z1.d, z3.h, z3.h
+    cbnz            w12, .Loop_sse_pp_16x\h
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+.endm
+
+SSE_PP_16xN 16
+SSE_PP_16xN 32
+
+.macro SSE_PP_32xN h
+function  PFX(pixel_sse_pp_32x\h\()_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, \h
+.Loop_sse_pp_32x\h:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    uabd            v2.8h, v16.8h, v20.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v17.8h, v21.8h
+    udot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    uabd            v2.8h, v18.8h, v22.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v19.8h, v23.8h
+    udot            z1.d, z3.h, z3.h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_pp_32x\h
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+.endm
+
+SSE_PP_32xN 32
+SSE_PP_32xN 64
+
+function PFX(pixel_sse_pp_64x64_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #64
+.Loop_sse_pp_64x1:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    uabd            v2.8h, v16.8h, v20.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v17.8h, v21.8h
+    udot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    uabd            v2.8h, v18.8h, v22.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v19.8h, v23.8h
+    udot            z1.d, z3.h, z3.h
+
+    ldp             q16, q17, x0, #64
+    ldp             q20, q21, x2, #64
+    uabd            v2.8h, v16.8h, v20.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v17.8h, v21.8h
+    udot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19,  x0, #96
+    ldp             q22, q23, x2, #96
+    uabd            v2.8h, v18.8h, v22.8h
+    udot            z0.d, z2.h, z2.h
+    uabd            v3.8h, v19.8h, v23.8h
+    udot            z1.d, z3.h, z3.h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_pp_64x1
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+#endif // HIGH_BIT_DEPTH
+
+.macro SSE_SS_4x2
+    ldr             d16, x0
+    ldr             d17, x2
+    ldr             d18, x0, x1
+    ldr             d19, x2, x3
+    sub             v2.4h, v16.4h, v17.4h
+    sub             v3.4h, v18.4h, v19.4h
+    sdot            z0.d, z2.h, z2.h
+    sdot            z0.d, z3.h, z3.h
+.endm
+
+function PFX(pixel_sse_ss_4x4_sve)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    SSE_SS_4x2
+    add             x0, x0, x1, lsl #1
+    add             x2, x2, x3, lsl #1
+    SSE_SS_4x2
+
+    fmov            w0, s0
+    ret
+endfunc
+
+function PFX(pixel_sse_ss_8x8_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+.rept 4
+    ld1             {v16.8h}, x0, x1
+    ld1             {v17.8h}, x2, x3
+    sub             v2.8h, v16.8h, v17.8h
+    sdot            z0.d, z2.h, z2.h
+    ld1             {v18.8h}, x0, x1
+    ld1             {v19.8h}, x2, x3
+    sub             v3.8h, v18.8h, v19.8h
+    sdot            z1.d, z3.h, z3.h
+.endr
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            w0, s0
+    ret
+endfunc
+
+function PFX(pixel_sse_ss_16x16_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #16
+.Loop_sse_ss_16:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0, x1
+    ld1             {v18.8h-v19.8h}, x2, x3
+    sub             v2.8h, v16.8h, v18.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v17.8h, v19.8h
+    sdot            z1.d, z3.h, z3.h
+    cbnz            w12, .Loop_sse_ss_16
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_sse_ss_32x32_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #32
+.Loop_sse_ss_32:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    sub             v2.8h, v16.8h, v20.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v17.8h, v21.8h
+    sdot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    sub             v2.8h, v18.8h, v22.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v19.8h, v23.8h
+    sdot            z1.d, z3.h, z3.h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_ss_32
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+
+function PFX(pixel_sse_ss_64x64_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #64
+.Loop_sse_ss_64:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    sub             v2.8h, v16.8h, v20.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v17.8h, v21.8h
+    sdot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    sub             v2.8h, v18.8h, v22.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v19.8h, v23.8h
+    sdot            z1.d, z3.h, z3.h
+
+    ldp             q16, q17, x0, #64
+    ldp             q20, q21, x2, #64
+    sub             v2.8h, v16.8h, v20.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v17.8h, v21.8h
+    sdot            z1.d, z3.h, z3.h
+
+    ldp             q18, q19,  x0, #96
+    ldp             q22, q23, x2, #96
+    sub             v2.8h, v18.8h, v22.8h
+    sdot            z0.d, z2.h, z2.h
+    sub             v3.8h, v19.8h, v23.8h
+    sdot            z1.d, z3.h, z3.h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_ss_64
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_4x4_sve)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+
+    ldr             d16, x0
+    ldr             d17, x0, x1
+    sdot            z0.d, z16.h, z16.h
+    sdot            z0.d, z17.h, z17.h
+    add             x0, x0, x1, lsl #1
+    ldr             d16, x0
+    ldr             d17, x0, x1
+    sdot            z0.d, z16.h, z16.h
+    sdot            z0.d, z17.h, z17.h
+
+    fmov            w0, s0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_8x8_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+
+.rept 4
+    ld1             {v16.8h}, x0, x1
+    sdot            z0.d, z16.h, z16.h
+    ld1             {v17.8h}, x0, x1
+    sdot            z1.d, z17.h, z17.h
+.endr
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            w0, s0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_16x16_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+
+    mov             w12, #16
+.Loop_ssd_s_16:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0, x1
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+    cbnz            w12, .Loop_ssd_s_16
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_32x32_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+
+    mov             w12, #32
+.Loop_ssd_s_32:
+    sub             w12, w12, #1
+
+    ldp             q16, q17, x0
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+    ldp             q16, q17, x0, #32
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+
+    add             x0, x0, x1
+    cbnz            w12, .Loop_ssd_s_32
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_64x64_sve)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+
+    mov             w12, #64
+.Loop_ssd_s_64:
+    sub             w12, w12, #1
+
+    ldp             q16, q17, x0
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+    ldp             q16, q17, x0, #32
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+    ldp             q16, q17, x0, #64
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+    ldp             q16, q17,  x0, #96
+    sdot            z0.d, z16.h, z16.h
+    sdot            z1.d, z17.h, z17.h
+
+    add             x0, x0, x1
+    cbnz            w12, .Loop_ssd_s_64
+
+    add             v0.2d, v0.2d, v1.2d
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc

x265-4.1.tar/source/common/aarch64/ssd-a.S -> x265-4.2.tar/source/common/aarch64/ssd-a.S Changed

@@ -3,6 +3,7 @@
  *
  * Authors: Sebastian Pop <spop@amazon.com>
  *          Hari Limaye <hari.limaye@arm.com>
+ *          Gerda Zsejke More <gerdazsejke.more@arm.com>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
@@ -35,6 +36,7 @@
 
 .text
 
+#if !HIGH_BIT_DEPTH
 // Fully unrolled.
 .macro SSE_PP_4xN h
 function PFX(pixel_sse_pp_4x\h\()_neon)
@@ -403,3 +405,561 @@
     add             v0.4s, v0.4s, v1.4s
     ret_v0_w0
 endfunc
+
+function PFX(pixel_ssd_s_64x64_neon)
+    add             x1, x1, x1
+    sub             x1, x1, #64
+    sub             x3, x3, #64
+
+    mov             w12, #32
+    movi            v0.16b, #0
+    movi            v1.16b, #0
+.Loop_ssd_ss_64:
+    sub             w12, w12, #1
+.rept 2
+    ld1             {v16.16b-v19.16b}, x0, #64
+    ld1             {v20.16b-v23.16b}, x0, x1
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal2          v1.4s, v16.8h, v16.8h
+    smlal           v0.4s, v17.4h, v17.4h
+    smlal2          v1.4s, v17.8h, v17.8h
+    smlal           v0.4s, v18.4h, v18.4h
+    smlal2          v1.4s, v18.8h, v18.8h
+    smlal           v0.4s, v19.4h, v19.4h
+    smlal2          v1.4s, v19.8h, v19.8h
+    smlal           v0.4s, v20.4h, v20.4h
+    smlal2          v1.4s, v20.8h, v20.8h
+    smlal           v0.4s, v21.4h, v21.4h
+    smlal2          v1.4s, v21.8h, v21.8h
+    smlal           v0.4s, v22.4h, v22.4h
+    smlal2          v1.4s, v22.8h, v22.8h
+    smlal           v0.4s, v23.4h, v23.4h
+    smlal2          v1.4s, v23.8h, v23.8h
+.endr
+    cbnz            w12, .Loop_ssd_ss_64
+    add             v0.4s, v0.4s, v1.4s
+    ret_v0_w0
+endfunc
+
+#else // HIGH_BIT_DEPTH
+
+.macro SSE_PP_4x2
+    ldr             d16, x0
+    ldr             d17, x2
+    ldr             d18, x0, x1
+    ldr             d19, x2, x3
+    uabd            v2.4h, v16.4h, v17.4h
+    uabd            v3.4h, v18.4h, v19.4h
+    umlal           v0.4s, v2.4h, v2.4h
+    umlal           v0.4s, v3.4h, v3.4h
+.endm
+
+.macro SSE_PP_4xN h
+function PFX(pixel_sse_pp_4x\h\()_neon)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+.rept (\h / 2) - 1
+    SSE_PP_4x2
+    add             x0, x0, x1, lsl #1
+    add             x2, x2, x3, lsl #1
+.endr
+    SSE_PP_4x2
+
+    ret_v0_w0
+endfunc
+.endm
+
+SSE_PP_4xN 4
+SSE_PP_4xN 8
+
+.macro SSE_PP_8xN h
+function PFX(pixel_sse_pp_8x\h\()_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+.rept \h
+    ld1             {v16.8h}, x0, x1
+    ld1             {v17.8h}, x2, x3
+    uabd            v2.8h, v16.8h, v17.8h
+    umlal           v0.4s, v2.4h, v2.4h
+    umlal2          v1.4s, v2.8h, v2.8h
+.endr
+    add             v0.4s, v0.4s, v1.4s
+
+    ret_v0_w0
+endfunc
+.endm
+
+SSE_PP_8xN 8
+SSE_PP_8xN 16
+
+.macro SSE_PP_16xN h
+function PFX(pixel_sse_pp_16x\h\()_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+.rept \h
+    ld1             {v16.8h-v17.8h}, x0, x1
+    ld1             {v18.8h-v19.8h}, x2, x3
+    uabd            v2.8h, v16.8h, v18.8h
+    umlal           v0.4s, v2.4h, v2.4h
+    umlal2          v1.4s, v2.8h, v2.8h
+    uabd            v3.8h, v17.8h, v19.8h
+    umlal           v0.4s, v3.4h, v3.4h
+    umlal2          v1.4s, v3.8h, v3.8h
+.endr
+
+.if \h == 16
+    add             v0.4s, v0.4s, v1.4s
+    addv            s0, v0.4s
+    fmov            w0, s0
+.else
+    uaddlv          d0, v0.4s
+    uaddlv          d1, v1.4s
+    add             d0, d0, d1
+    fmov            x0, d0
+.endif
+
+    ret
+endfunc
+.endm
+
+SSE_PP_16xN 16
+SSE_PP_16xN 32
+
+.macro SSE_PP_32xN h
+function  PFX(pixel_sse_pp_32x\h\()_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, \h
+.Loop_sse_pp_32x\h:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    uabd            v2.8h, v16.8h, v20.8h
+    umlal           v0.4s, v2.4h, v2.4h
+    umlal2          v1.4s, v2.8h, v2.8h
+    uabd            v3.8h, v17.8h, v21.8h
+    umlal           v0.4s, v3.4h, v3.4h
+    umlal2          v1.4s, v3.8h, v3.8h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    uabd            v2.8h, v18.8h, v22.8h
+    umlal           v0.4s, v2.4h, v2.4h
+    umlal2          v1.4s, v2.8h, v2.8h
+    uabd            v3.8h, v19.8h, v23.8h
+    umlal           v0.4s, v3.4h, v3.4h
+    umlal2          v1.4s, v3.8h, v3.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_pp_32x\h
+
+    uaddlv          d0, v0.4s
+    uaddlv          d1, v1.4s
+    add             d0, d0, d1
+    fmov            x0, d0
+    ret
+endfunc
+.endm
+
+SSE_PP_32xN 32
+SSE_PP_32xN 64
+
+function PFX(pixel_sse_pp_64x64_neon)
+    mov             w12, #64
+
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    movi            v2.4s, #0
+    movi            v3.4s, #0
+
+    add             x1, x1, x1
+    add             x3, x3, x3
+.Loop_sse_pp_64x1:
+    sub             w12, w12, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v20.8h-v21.8h}, x2
+    uabd            v4.8h, v16.8h, v20.8h
+    umlal           v0.4s, v4.4h, v4.4h
+    umlal2          v1.4s, v4.8h, v4.8h
+    uabd            v5.8h, v17.8h, v21.8h
+    umlal           v0.4s, v5.4h, v5.4h
+    umlal2          v1.4s, v5.8h, v5.8h
+
+    ldp             q18, q19, x0, #32
+    ldp             q22, q23, x2, #32
+    uabd            v6.8h, v18.8h, v22.8h
+    umlal           v2.4s, v6.4h, v6.4h
+    umlal2          v3.4s, v6.8h, v6.8h
+    uabd            v7.8h, v19.8h, v23.8h
+    umlal           v2.4s, v7.4h, v7.4h
+    umlal2          v3.4s, v7.8h, v7.8h
+
+    ldp             q16, q17, x0, #64
+    ldp             q20, q21, x2, #64
+    uabd            v4.8h, v16.8h, v20.8h
+    umlal           v0.4s, v4.4h, v4.4h
+    umlal2          v1.4s, v4.8h, v4.8h
+    uabd            v5.8h, v17.8h, v21.8h
+    umlal           v0.4s, v5.4h, v5.4h
+    umlal2          v1.4s, v5.8h, v5.8h
+
+    ldp             q18, q19, x0, #96
+    ldp             q22, q23, x2, #96
+    uabd            v6.8h, v18.8h, v22.8h
+    umlal           v2.4s, v6.4h, v6.4h
+    umlal2          v3.4s, v6.8h, v6.8h
+    uabd            v7.8h, v19.8h, v23.8h
+    umlal           v2.4s, v7.4h, v7.4h
+    umlal2          v3.4s, v7.8h, v7.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_pp_64x1
+
+    uaddlv          d0, v0.4s
+    uaddlv          d1, v1.4s
+    add             d0, d0, d1
+    uaddlv          d2, v2.4s
+    uaddlv          d3, v3.4s
+    add             d2, d2, d3
+
+    add             d0, d0, d2
+    fmov            x0, d0
+    ret
+endfunc
+
+.macro SSE_SS_4x2
+    ldr             d16, x0
+    ldr             d17, x2
+    ldr             d18, x0, x1
+    ldr             d19, x2, x3
+    sub             v2.4h, v16.4h, v17.4h
+    sub             v3.4h, v18.4h, v19.4h
+    smlal           v0.4s, v2.4h, v2.4h
+    smlal           v0.4s, v3.4h, v3.4h
+.endm
+
+function PFX(pixel_sse_ss_4x4_neon)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    SSE_SS_4x2
+    add             x0, x0, x1, lsl 1
+    add             x2, x2, x3, lsl 1
+    SSE_SS_4x2
+
+    ret_v0_w0
+endfunc
+
+function PFX(pixel_sse_ss_8x8_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+.rept 8
+    ld1             {v16.8h}, x0, x1
+    ld1             {v17.8h}, x2, x3
+    sub             v2.8h, v16.8h, v17.8h
+    smlal           v0.4s, v2.4h, v2.4h
+    smlal2          v1.4s, v2.8h, v2.8h
+.endr
+    add             v0.4s, v0.4s, v1.4s
+    ret_v0_w0
+endfunc
+
+function PFX(pixel_sse_ss_16x16_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+    mov             w12, #16
+.Loop_sse_ss_16:
+    sub             w12, w12, #1
+    ld1             {v16.8h-v17.8h}, x0, x1
+    ld1             {v18.8h-v19.8h}, x2, x3
+    sub             v2.8h, v16.8h, v18.8h
+    smlal           v0.4s, v2.4h, v2.4h
+    smlal2          v1.4s, v2.8h, v2.8h
+    sub             v3.8h, v17.8h, v19.8h
+    smlal           v0.4s, v3.4h, v3.4h
+    smlal2          v1.4s, v3.8h, v3.8h
+    cbnz            w12, .Loop_sse_ss_16
+
+    add             v0.4s, v0.4s, v1.4s
+    uaddlv          d0, v0.4s
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_sse_ss_32x32_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    movi            v2.4s, #0
+    movi            v3.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+    mov             w12, #32
+.Loop_sse_ss_32:
+    sub             w12, w12, #1
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v18.8h-v19.8h}, x2
+    sub             v4.8h, v16.8h, v18.8h
+    smlal           v0.4s, v4.4h, v4.4h
+    smlal2          v1.4s, v4.8h, v4.8h
+    sub             v5.8h, v17.8h, v19.8h
+    smlal           v0.4s, v5.4h, v5.4h
+    smlal2          v1.4s, v5.8h, v5.8h
+
+    ldp             q16, q17, x0, #32
+    ldp             q18, q19, x2, #32
+    sub             v6.8h, v16.8h, v18.8h
+    smlal           v2.4s, v6.4h, v6.4h
+    smlal2          v3.4s, v6.8h, v6.8h
+    sub             v7.8h, v17.8h, v19.8h
+    smlal           v2.4s, v7.4h, v7.4h
+    smlal2          v3.4s, v7.8h, v7.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w12, .Loop_sse_ss_32
+
+    uaddlp          v0.2d, v0.4s
+    uadalp          v0.2d, v1.4s
+    uadalp          v0.2d, v2.4s
+    uadalp          v0.2d, v3.4s
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_sse_ss_64x64_neon)
+    movi            v0.4s, #0
+    movi            v31.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #2
+.Loop_sse_ss_64x32:
+    sub             w12, w12, #1
+    movi            v1.4s, #0
+    movi            v2.4s, #0
+    movi            v3.4s, #0
+    movi            v4.4s, #0
+    movi            v5.4s, #0
+    movi            v6.4s, #0
+    movi            v7.4s, #0
+    movi            v8.4s, #0
+    mov             w11, #32
+.Loop_sse_ss_64x1:
+    sub             w11, w11, #1
+
+    ld1             {v16.8h-v17.8h}, x0
+    ld1             {v18.8h-v19.8h}, x2
+    sub             v20.8h, v16.8h, v18.8h
+    smlal           v1.4s, v20.4h, v20.4h
+    smlal2          v2.4s, v20.8h, v20.8h
+    sub             v21.8h, v17.8h, v19.8h
+    smlal           v1.4s, v21.4h, v21.4h
+    smlal2          v2.4s, v21.8h, v21.8h
+
+    ldp             q16, q17, x0, #32
+    ldp             q18, q19, x2, #32
+    sub             v22.8h, v16.8h, v18.8h
+    smlal           v3.4s, v22.4h, v22.4h
+    smlal2          v4.4s, v22.8h, v22.8h
+    sub             v23.8h, v17.8h, v19.8h
+    smlal           v3.4s, v23.4h, v23.4h
+    smlal2          v4.4s, v23.8h, v23.8h
+
+    ldp             q16, q17, x0, #64
+    ldp             q18, q19, x2, #64
+    sub             v20.8h, v16.8h, v18.8h
+    smlal           v5.4s, v20.4h, v20.4h
+    smlal2          v6.4s, v20.8h, v20.8h
+    sub             v21.8h, v17.8h, v19.8h
+    smlal           v5.4s, v21.4h, v21.4h
+    smlal2          v6.4s, v21.8h, v21.8h
+
+    ldp             q16, q17, x0, #96
+    ldp             q18, q19, x2, #96
+    sub             v22.8h, v16.8h, v18.8h
+    smlal           v7.4s, v22.4h, v22.4h
+    smlal2          v8.4s, v22.8h, v22.8h
+    sub             v23.8h, v17.8h, v19.8h
+    smlal           v7.4s, v23.4h, v23.4h
+    smlal2          v8.4s, v23.8h, v23.8h
+
+    add             x0, x0, x1
+    add             x2, x2, x3
+    cbnz            w11, .Loop_sse_ss_64x1
+
+    uadalp          v0.2d, v1.4s
+    uadalp          v0.2d, v2.4s
+    uadalp          v0.2d, v3.4s
+    uadalp          v0.2d, v4.4s
+    uadalp          v0.2d, v5.4s
+    uadalp          v0.2d, v6.4s
+    uadalp          v0.2d, v7.4s
+    uadalp          v0.2d, v8.4s
+    cbnz            w12, .Loop_sse_ss_64x32
+
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_4x4_neon)
+    movi            v0.4s, #0
+    add             x1, x1, x1
+
+    ldr             d16, x0
+    ldr             d17, x0, x1
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal           v0.4s, v17.4h, v17.4h
+    add             x0, x0, x1, lsl 1
+    ldr             d16, x0
+    ldr             d17, x0, x1
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal           v0.4s, v17.4h, v17.4h
+
+    ret_v0_w0
+endfunc
+
+function PFX(pixel_ssd_s_8x8_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+.rept 8
+    ld1             {v16.8h}, x0, x1
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal2          v1.4s, v16.8h, v16.8h
+.endr
+    add             v0.4s, v0.4s, v1.4s
+    ret_v0_w0
+endfunc
+
+function PFX(pixel_ssd_s_16x16_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    add             x1, x1, x1
+
+    mov             w12, #16
+.Loop_ssd_s_16:
+    sub             w12, w12, #1
+    ld1             {v16.8h-v17.8h}, x0, x1
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal2          v1.4s, v16.8h, v16.8h
+    smlal           v0.4s, v17.4h, v17.4h
+    smlal2          v1.4s, v17.8h, v17.8h
+    cbnz            w12, .Loop_ssd_s_16
+
+    add             v0.4s, v0.4s, v1.4s
+    uaddlv          d0, v0.4s
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_32x32_neon)
+    movi            v0.4s, #0
+    movi            v1.4s, #0
+    movi            v2.4s, #0
+    movi            v3.4s, #0
+    add             x1, x1, x1
+
+    mov             w12, #32
+.Loop_ssd_s_32:
+    sub             w12, w12, #1
+
+    ldp             q16, q17, x0
+    smlal           v0.4s, v16.4h, v16.4h
+    smlal2          v1.4s, v16.8h, v16.8h
+    smlal           v0.4s, v17.4h, v17.4h
+    smlal2          v1.4s, v17.8h, v17.8h
+
+    ldp             q16, q17, x0, #32
+    smlal           v2.4s, v16.4h, v16.4h
+    smlal2          v3.4s, v16.8h, v16.8h
+    smlal           v2.4s, v17.4h, v17.4h
+    smlal2          v3.4s, v17.8h, v17.8h
+
+    add             x0, x0, x1
+    cbnz            w12, .Loop_ssd_s_32
+
+    uaddlp          v0.2d, v0.4s
+    uadalp          v0.2d, v1.4s
+    uadalp          v0.2d, v2.4s
+    uadalp          v0.2d, v3.4s
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+function PFX(pixel_ssd_s_64x64_neon)
+    movi            v0.4s, #0
+    movi            v31.4s, #0
+    add             x1, x1, x1
+    add             x3, x3, x3
+
+    mov             w12, #2
+.Loop_ssd_s_64x32:
+    sub             w12, w12, #1
+    movi            v1.4s, #0
+    movi            v2.4s, #0
+    movi            v3.4s, #0
+    movi            v4.4s, #0
+    mov             w11, #32
+.Loop_ssd_s_64x1:
+    sub             w11, w11, #1
+
+    ldp             q16, q17, x0
+    smlal           v1.4s, v16.4h, v16.4h
+    smlal2          v2.4s, v16.8h, v16.8h
+    smlal           v1.4s, v17.4h, v17.4h
+    smlal2          v2.4s, v17.8h, v17.8h
+
+    ldp             q16, q17, x0, #32
+    smlal           v3.4s, v16.4h, v16.4h
+    smlal2          v4.4s, v16.8h, v16.8h
+    smlal           v3.4s, v17.4h, v17.4h
+    smlal2          v4.4s, v17.8h, v17.8h
+
+    ldp             q16, q17, x0, #64
+    smlal           v1.4s, v16.4h, v16.4h
+    smlal2          v2.4s, v16.8h, v16.8h
+    smlal           v1.4s, v17.4h, v17.4h
+    smlal2          v2.4s, v17.8h, v17.8h
+
+    ldp             q16, q17, x0, #96
+    smlal           v3.4s, v16.4h, v16.4h
+    smlal2          v4.4s, v16.8h, v16.8h
+    smlal           v3.4s, v17.4h, v17.4h
+    smlal2          v4.4s, v17.8h, v17.8h
+
+    add             x0, x0, x1
+    cbnz            w11, .Loop_ssd_s_64x1
+
+    uadalp          v0.2d, v1.4s
+    uadalp          v0.2d, v2.4s
+    uadalp          v0.2d, v3.4s
+    uadalp          v0.2d, v4.4s
+    cbnz            w12, .Loop_ssd_s_64x32
+
+    addp            d0, v0.2d
+    fmov            x0, d0
+    ret
+endfunc
+
+#endif // !HIGH_BIT_DEPTH

x265-4.1.tar/source/common/bitstream.cpp -> x265-4.2.tar/source/common/bitstream.cpp Changed

x265-4.1.tar/source/common/common.h -> x265-4.2.tar/source/common/common.h Changed

@@ -39,6 +39,7 @@
 #include <stdint.h>
 #include <memory.h>
 #include <assert.h>
+#include <stdlib.h>
 
 #include "x265.h"
 
@@ -150,6 +151,7 @@
 #endif
 
 #define MAX_UINT        0xFFFFFFFFU // max. value of unsigned 32-bit integer
+#define MAX_UINT64      0xFFFFFFFFFFFFFFFFULL // max. value of unsigned 64-bit integer
 #define MAX_INT         2147483647  // max. value of signed 32-bit integer
 #define MAX_INT64       0x7FFFFFFFFFFFFFFFLL  // max. value of signed 64-bit integer
 #define MAX_DOUBLE      1.7e+308    // max. value of double-type value
@@ -231,7 +233,7 @@
         var = (type*)x265_malloc(sizeof(type) * (count)); \
         if (!var) \
         { \
-            x265_log(NULL, X265_LOG_ERROR, "malloc of size %d failed\n", sizeof(type) * (count)); \
+            x265_log(NULL, X265_LOG_ERROR, "malloc of size %llu failed\n", sizeof(type) * (count)); \
             goto fail; \
         } \
     }
@@ -242,7 +244,7 @@
             memset((void*)var, 0, sizeof(type) * (count)); \
         else \
         { \
-            x265_log(NULL, X265_LOG_ERROR, "malloc of size %d failed\n", sizeof(type) * (count)); \
+            x265_log(NULL, X265_LOG_ERROR, "malloc of size %llu failed\n", sizeof(type) * (count)); \
             goto fail; \
         } \
     }
@@ -350,6 +352,10 @@
 
 #define MAX_MCSTF_TEMPORAL_WINDOW_LENGTH 8
 
+#define MAX_NUM_PUS_PER_CTU      593   // Maximum number of PUs in a 64x64 CTU
+#define MAX_NUM_PU_SIZES         24    // Number of distinct PU sizes in a 64x64 CTU
+#define MIN_TME_THREADS          32    // Recommended number of threads for ThreadedME
+
 namespace X265_NS {
 
 enum { SAO_NUM_OFFSET = 4 };

x265-4.1.tar/source/common/cpu.cpp -> x265-4.2.tar/source/common/cpu.cpp Changed

@@ -62,7 +62,9 @@
 #endif // if X265_ARCH_ARM
 
 namespace X265_NS {
+#if X265_ARCH_X86
 static bool enable512 = false;
+#endif
 const cpu_name_t cpu_names =
 {
 #if X265_ARCH_X86
@@ -121,9 +123,16 @@
 #if defined(HAVE_NEON_I8MM)
     { "Neon_I8MM",       X265_CPU_NEON_I8MM },
 #endif
+#if defined(HAVE_SVE2_BITPERM)
+    { "SVE2_BitPerm",    X265_CPU_SVE2_BITPERM },
+#endif
 #elif X265_ARCH_POWER8
     { "Altivec",         X265_CPU_ALTIVEC },
 
+#elif X265_ARCH_RISCV64
+    { "RVV",           X265_CPU_RVV },
+    { "Zbb",           X265_CPU_ZBB },
+
 #endif // if X265_ARCH_X86
     { "", 0 },
 };
@@ -352,7 +361,7 @@
 {
     int flags = 0;
 
-#if HAVE_ARMV6
+#if HAVE_ARMV6 && ENABLE_ASSEMBLY
     flags |= X265_CPU_ARMV6;
 
     // don't do this hack if compiled with -mfpu=neon
@@ -404,6 +413,21 @@
     return flags;
 }
 
+#elif X265_ARCH_RISCV64
+#include "riscv64/cpu.h"
+
+uint32_t cpu_detect(bool benableavx512)
+{
+    (void)benableavx512;
+    uint32_t flags = 0;
+
+#ifdef ENABLE_ASSEMBLY
+    flags = riscv64_cpu_detect();
+#endif
+
+    return flags;
+}
+
 #elif X265_ARCH_POWER8
 
 uint32_t cpu_detect(bool benableavx512)

x265-4.1.tar/source/common/cudata.cpp -> x265-4.2.tar/source/common/cudata.cpp Changed

@@ -1740,6 +1740,64 @@
     return count;
 }
 
+bool CUData::getMedianColMV(const CUData* colCU, const Frame* colPic, int list, int ref, MV& outMV) const
+{
+    int mvCount = 0;
+    int mvXMAX_NUM_PARTITIONS, mvYMAX_NUM_PARTITIONS;
+
+    for (uint32_t partIdx = 0; partIdx < colCU->m_numPartitions; partIdx++)
+    {
+        uint32_t absPartAddr = partIdx & TMVP_UNIT_MASK;
+        if (colCU->m_predModepartIdx == MODE_NONE || colCU->isIntra(absPartAddr))
+            continue;
+
+        int8_t refIdx = colCU->m_refIdxlistpartIdx;
+        if (refIdx < 0)
+            continue;
+
+        MV rawMv = colCU->m_mvlistpartIdx;
+
+        int colPOC = colPic->m_encData->m_slice->m_poc;
+        int colRefPOC = colPic->m_encData->m_slice->m_refPOCListlistrefIdx;
+
+        int curPOC = m_slice->m_poc;
+        int curRefPOC = this->m_slice->m_refPOCListlistref;
+
+        MV scaledMv = scaleMvByPOCDist(rawMv, curPOC, curRefPOC, colPOC, colRefPOC);
+
+        if (mvCount >= MAX_NUM_PARTITIONS)
+            break;
+
+        mvXmvCount = scaledMv.x;
+        mvYmvCount = scaledMv.y;
+        mvCount++;
+    }
+
+    if (mvCount == 0)
+        return false;
+
+    size_t mid = mvCount >> 1;
+
+    std::nth_element(mvX, mvX + mid, mvX + mvCount);
+    std::nth_element(mvY, mvY + mid, mvY + mvCount);
+
+    if (mvCount & 1)
+    {
+        outMV.x = mvXmid;
+        outMV.y = mvYmid;
+    }
+    else
+    {
+        int lowerMaxX = *std::max_element(mvX, mvX + mid);
+        int lowerMaxY = *std::max_element(mvY, mvY + mid);
+
+        outMV.x = (lowerMaxX + mvXmid) >> 1;
+        outMV.y = (lowerMaxY + mvYmid) >> 1;
+    }
+
+    return true;
+}
+
 // Create the PMV list. Called for each reference index.
 #if (ENABLE_MULTIVIEW || ENABLE_SCC_EXT)
 int CUData::getPMV(InterNeighbourMV* neighbours, uint32_t picList, uint32_t refIdx, MV* amvpCand, MV* pmv, uint32_t puIdx, uint32_t absPartIdx) const

x265-4.1.tar/source/common/cudata.h -> x265-4.2.tar/source/common/cudata.h Changed

x265-4.1.tar/source/common/dct.cpp -> x265-4.2.tar/source/common/dct.cpp Changed

x265-4.1.tar/source/common/frame.cpp -> x265-4.2.tar/source/common/frame.cpp Changed

@@ -36,6 +36,7 @@
     m_reconRowFlag = NULL;
     m_reconColCount = NULL;
     m_countRefEncoders = 0;
+    m_ctuMEFlags = NULL;
     m_encData = NULL;
     for (int i = 0; i < NUM_RECON_VERSION; i++)
         m_reconPici = NULL;
@@ -179,9 +180,10 @@
     {
         X265_CHECK((m_reconColCount == NULL), "m_reconColCount was initialized");
         m_numRows = (m_fencPic->m_picHeight + param->maxCUSize - 1)  / param->maxCUSize;
+        int32_t m_numCols = (m_fencPic->m_picWidth + param->maxCUSize - 1)  / param->maxCUSize;
         m_reconRowFlag = new ThreadSafeIntegerm_numRows;
         m_reconColCount = new ThreadSafeIntegerm_numRows;
-
+        m_ctuMEFlags = new ThreadSafeIntegerm_numRows * m_numCols;
         if (quantOffsets)
         {
             int32_t cuCount = (param->rc.qgSize == 8) ? m_lowres.maxBlocksInRowFullRes * m_lowres.maxBlocksInColFullRes :
@@ -358,6 +360,12 @@
         m_reconColCount = NULL;
     }
 
+    if (m_ctuMEFlags)
+    {
+        delete m_ctuMEFlags;
+        m_ctuMEFlags = NULL;
+    }
+
     if (m_quantOffsets)
     {
         delete m_quantOffsets;

x265-4.1.tar/source/common/frame.h -> x265-4.2.tar/source/common/frame.h Changed

x265-4.1.tar/source/common/framedata.cpp -> x265-4.2.tar/source/common/framedata.cpp Changed

x265-4.1.tar/source/common/loopfilter.cpp -> x265-4.2.tar/source/common/loopfilter.cpp Changed

@@ -28,6 +28,8 @@
 
 #define PIXEL_MIN 0
 
+using namespace X265_NS;
+
 namespace {
 
 static void calSign(int8_t *dst, const pixel *src1, const pixel *src2, const int endX)
@@ -152,26 +154,94 @@
     }
 }
 
-/* Deblocking of one line/column for the chrominance component
-* \param src     pointer to picture data
-* \param offset  offset value for picture data
-* \param tc      tc value
-* \param maskP   indicator to disable filtering on partP
-* \param maskQ   indicator to disable filtering on partQ */
-static void pelFilterChroma_c(pixel* src, intptr_t srcStep, intptr_t offset, int32_t tc, int32_t maskP, int32_t maskQ)
+void pelFilterChroma_V_c(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc,
+                         int32_t maskP, int32_t maskQ)
 {
-    for (int32_t i = 0; i < UNIT_SIZE; i++, src += srcStep)
-    {
-        int16_t m4 = (int16_t)src0;
-        int16_t m3 = (int16_t)src-offset;
-        int16_t m5 = (int16_t)srcoffset;
-        int16_t m2 = (int16_t)src-offset * 2;
-
-        int32_t delta = x265_clip3(-tc, tc, ((((m4 - m3) * 4) + m2 - m5 + 4) >> 3));
-        src-offset  = x265_clip(m3 + (delta & maskP));
-        src0        = x265_clip(m4 - (delta & maskQ));
-    }
+    X265_CHECK(offset == 1, "Offset value must be 1 for Chroma Vertical\n");
+
+    (void)offset;
+
+    int16_t m2 = (int16_t)src0 * srcStep - 2;
+    int16_t m3 = (int16_t)src0 * srcStep - 1;
+    int16_t m4 = (int16_t)src0 * srcStep + 0;
+    int16_t m5 = (int16_t)src0 * srcStep + 1;
+
+    int32_t delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src0 * srcStep - 1 = x265_clip(m3 + (delta & maskP));
+    src0 * srcStep + 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src1 * srcStep - 2;
+    m3 = (int16_t)src1 * srcStep - 1;
+    m4 = (int16_t)src1 * srcStep + 0;
+    m5 = (int16_t)src1 * srcStep + 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src1 * srcStep - 1 = x265_clip(m3 + (delta & maskP));
+    src1 * srcStep + 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src2 * srcStep - 2;
+    m3 = (int16_t)src2 * srcStep - 1;
+    m4 = (int16_t)src2 * srcStep + 0;
+    m5 = (int16_t)src2 * srcStep + 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src2 * srcStep - 1 = x265_clip(m3 + (delta & maskP));
+    src2 * srcStep + 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src3 * srcStep - 2;
+    m3 = (int16_t)src3 * srcStep - 1;
+    m4 = (int16_t)src3 * srcStep + 0;
+    m5 = (int16_t)src3 * srcStep + 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src3 * srcStep - 1 = x265_clip(m3 + (delta & maskP));
+    src3 * srcStep + 0 = x265_clip(m4 - (delta & maskQ));
 }
+
+void pelFilterChroma_H_c(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc,
+                         int32_t maskP, int32_t maskQ)
+{
+    X265_CHECK(srcStep == 1, "srcStep value must be 1 for Chroma Horizontal\n");
+
+    (void)srcStep;
+
+    int16_t m2 = (int16_t)src0 - offset * 2;
+    int16_t m3 = (int16_t)src0 - offset * 1;
+    int16_t m4 = (int16_t)src0 + offset * 0;
+    int16_t m5 = (int16_t)src0 + offset * 1;
+
+    int32_t delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src0 - offset * 1 = x265_clip(m3 + (delta & maskP));
+    src0 + offset * 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src1 - offset * 2;
+    m3 = (int16_t)src1 - offset * 1;
+    m4 = (int16_t)src1 + offset * 0;
+    m5 = (int16_t)src1 + offset * 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src1 - offset * 1 = x265_clip(m3 + (delta & maskP));
+    src1 + offset * 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src2 - offset * 2;
+    m3 = (int16_t)src2 - offset * 1;
+    m4 = (int16_t)src2 + offset * 0;
+    m5 = (int16_t)src2 + offset * 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src2 - offset * 1 = x265_clip(m3 + (delta & maskP));
+    src2 + offset * 0 = x265_clip(m4 - (delta & maskQ));
+
+    m2 = (int16_t)src3 - offset * 2;
+    m3 = (int16_t)src3 - offset * 1;
+    m4 = (int16_t)src3 + offset * 0;
+    m5 = (int16_t)src3 + offset * 1;
+
+    delta = x265_clip3(-tc, tc, ((m4 - m3) * 4 + m2 - m5 + 4) >> 3);
+    src3 - offset * 1 = x265_clip(m3 + (delta & maskP));
+    src3 + offset * 0 = x265_clip(m4 - (delta & maskQ));
+}
+
 }
 
 namespace X265_NS {
@@ -190,7 +260,7 @@
     // C code is same for EDGE_VER and EDGE_HOR only asm code is different
     p.pelFilterLumaStrong0 = pelFilterLumaStrong_c;
     p.pelFilterLumaStrong1 = pelFilterLumaStrong_c;
-    p.pelFilterChroma0     = pelFilterChroma_c;
-    p.pelFilterChroma1     = pelFilterChroma_c;
+    p.pelFilterChroma0     = pelFilterChroma_V_c;
+    p.pelFilterChroma1     = pelFilterChroma_H_c;
 }
 }

x265-4.1.tar/source/common/lowres.cpp -> x265-4.2.tar/source/common/lowres.cpp Changed

x265-4.1.tar/source/common/param.cpp -> x265-4.2.tar/source/common/param.cpp Changed

@@ -423,6 +423,10 @@
     param->searchRangeForLayer1 = 3;
     param->searchRangeForLayer2 = 3;
 
+    /* Threaded ME */
+    param->tmeTaskBlockSize = 1;
+    param->tmeNumBufferRows = 10;
+
     /*Alpha Channel Encoding*/
     param->bEnableAlpha = 0;
     param->numScalableLayers = 1;
@@ -641,6 +645,7 @@
     }
     if (tune)
     {
+        param->tune = tune;
         if (!strcmp(tune, "psnr"))
         {
             param->rc.aqStrength = 0.0;
@@ -904,12 +909,16 @@
     bool bValueWasNull = !value;
     bool bExtraParams = false;
     char nameBuf64;
+#ifdef SVT_HEVC
     static int count;
+#endif
 
     if (!name)
         return X265_PARAM_BAD_NAME;
 
+#ifdef SVT_HEVC
     count++;
+#endif
     // skip -- prefix if provided
     if (name0 == '-' && name1 == '-')
         name += 2;
@@ -1518,6 +1527,7 @@
         }
 #endif
         OPT("frame-rc") p->bConfigRCFrame = atobool(value);
+        OPT("threaded-me") p->bThreadedME = atobool(value);
         else
             return X265_PARAM_BAD_NAME;
     }
@@ -1667,6 +1677,8 @@
 {
 #define CHECK(expr, msg) check_failed |= _confirm(param, expr, msg)
     int check_failed = 0; /* abort if there is a fatal configuration problem */
+    CHECK((uint64_t)param->sourceWidth * param->sourceHeight > 142606336ULL && !param->bAllowNonConformance,
+          "Input video resolution exceeds the maximum supported luma samples 142,606,336 (16384x8704) of Level 7.2.");
     CHECK(param->uhdBluray == 1 && (X265_DEPTH != 10 || param->internalCsp != 1 || param->interlaceMode != 0),
         "uhd-bd: bit depth, chroma subsample, source picture type must be 10, 4:2:0, progressive");
     CHECK(param->maxCUSize != 64 && param->maxCUSize != 32 && param->maxCUSize != 16,
@@ -1803,10 +1815,10 @@
           " smpte170m, smpte240m, linear, log100, log316, iec61966-2-4, bt1361e,"
           " iec61966-2-1, bt2020-10, bt2020-12, smpte-st-2084, smpte-st-428 or arib-std-b67");
     CHECK(param->vui.matrixCoeffs < 0
-          || param->vui.matrixCoeffs > 14
+          || param->vui.matrixCoeffs > 15
           || param->vui.matrixCoeffs == 3,
           "Matrix Coefficients must be unknown, bt709, fcc, bt470bg, smpte170m,"
-          " smpte240m, gbr, ycgco, bt2020nc, bt2020c, smpte-st-2085, chroma-nc, chroma-c or ictcp");
+          " smpte240m, gbr, ycgco, bt2020nc, bt2020c, smpte-st-2085, chroma-nc, chroma-c, ictcp or ipt-pq-c2");
     CHECK(param->vui.chromaSampleLocTypeTopField < 0
           || param->vui.chromaSampleLocTypeTopField > 5,
           "Chroma Sample Location Type Top Field must be 0-5");
@@ -1855,6 +1867,11 @@
         "Valid final VBV buffer emptiness must be a fraction 0 - 1, or size in kbits");
     CHECK(param->vbvEndFrameAdjust < 0,
         "Valid vbv-end-fr-adj must be a fraction 0 - 1");
+    if ((param->rc.vbvBufferSize > 0 || param->rc.vbvMaxBitrate > 0) && param->bThreadedME)
+    {
+        param->bThreadedME = 0;
+        x265_log(param, X265_LOG_WARNING, "VBV and threaded-me both enabled. Disabling threaded-me\n");
+    }
     CHECK(param->minVbvFullness < 0 && param->minVbvFullness > 100,
         "min-vbv-fullness must be a fraction 0 - 100");
     CHECK(param->maxVbvFullness < 0 && param->maxVbvFullness > 100,
@@ -1870,7 +1887,7 @@
     CHECK(param->rc.rateControlMode == X265_RC_CQP && param->rc.bStatRead,
           "Constant QP is incompatible with 2pass");
     CHECK(param->rc.bStrictCbr && (param->rc.bitrate <= 0 || param->rc.vbvBufferSize <=0),
-          "Strict-cbr cannot be applied without specifying target bitrate or vbv bufsize");
+          "Strict-cbr cannot be applied without specifying both target bitrate and vbv bufsize");
     CHECK(strlen(param->analysisSave) && (param->analysisSaveReuseLevel < 0 || param->analysisSaveReuseLevel > 10),
         "Invalid analysis save refine level. Value must be between 1 and 10 (inclusive)");
     CHECK(strlen(param->analysisLoad) && (param->analysisLoadReuseLevel < 0 || param->analysisLoadReuseLevel > 10),
@@ -1948,7 +1965,7 @@
             CHECK(param->hmeRangelevel < 0 || param->hmeRangelevel >= 32768,
                 "Search Range for HME levels must be between 0 and 32768");
     }
-#if !X86_64
+#if !X86_64 && !X265_ARCH_ARM64 && !X265_ARCH_RISCV64
     CHECK(param->searchMethod == X265_SEA && (param->sourceWidth > 840 || param->sourceHeight > 480),
         "SEA motion search does not support resolutions greater than 480p in 32 bit build");
 #endif
@@ -1971,11 +1988,6 @@
         param->bSingleSeiNal = 0;
         x265_log(param, X265_LOG_WARNING, "None of the SEI messages are enabled. Disabling Single SEI NAL\n");
     }
-    if (param->bEnableTemporalFilter && (param->frameNumThreads != 1))
-    {
-        param->bEnableTemporalFilter = 0;
-        x265_log(param, X265_LOG_WARNING, "MCSTF can be enabled with frame thread = 1 only. Disabling MCSTF\n");
-    }
     CHECK(param->confWinRightOffset < 0, "Conformance Window Right Offset must be 0 or greater");
     CHECK(param->confWinBottomOffset < 0, "Conformance Window Bottom Offset must be 0 or greater");
     CHECK(param->decoderVbvMaxRate < 0, "Invalid Decoder Vbv Maxrate. Value can not be less than zero");
@@ -2024,6 +2036,19 @@
         if (checkValid)
             param->bEnableSCC = 0;
     }
+    if (!!param->bEnableSCC)
+    {
+        if (param->bEnableRdRefine && param->bDynamicRefine)
+        {
+            param->bEnableRdRefine = 0;
+            x265_log(param, X265_LOG_WARNING, "Disabling rd-refine as it can not be used with scc and dynamic-refine\n");
+        }
+        if (param->bEnableRdRefine && param->interRefine > 0)
+        {
+            param->bEnableRdRefine = 0;
+            x265_log(param, X265_LOG_WARNING, "Disabling rd-refine as it can not be used with scc and inter-refine\n");
+        }
+    }
     CHECK(!!param->bEnableSCC&& param->rdLevel != 6, "Enabling scc extension in x265 requires rdlevel of 6 ");
 #endif
 
@@ -2073,6 +2098,9 @@
 
     x265_log(param, X265_LOG_INFO, "Coding QT: max CU size, min CU size : %d / %d\n", param->maxCUSize, param->minCUSize);
 
+    if (param->bThreadedME)
+        x265_log(param, X265_LOG_INFO, "ThreadedME: task block / buf rows   : %d / %d\n", param->tmeTaskBlockSize, param->tmeNumBufferRows);
+
     x265_log(param, X265_LOG_INFO, "Residual QT: max TU size, max depth : %d / %d inter / %d intra\n",
              param->maxTUSize, param->tuQTMaxInterDepth, param->tuQTMaxIntraDepth);
 
@@ -2200,6 +2228,8 @@
 #if ENABLE_HDR10_PLUS
     TOOLOPT(param->toneMapFile != NULL, "dhdr10-info");
 #endif
+    if(param->bEnableTemporalFilter)
+        TOOLOPT(param->bEnableTemporalFilter, "mcstf");
     x265_log(param, X265_LOG_INFO, "tools:%s\n", buf);
     fflush(stderr);
 }
@@ -2702,6 +2732,7 @@
     if (strlen(src->numaPools)) snprintf(dst->numaPools, X265_MAX_STRING_SIZE, "%s", src->numaPools);
     else dst->numaPools0 = 0;
 
+    dst->tune = src->tune;
     dst->bEnableWavefront = src->bEnableWavefront;
     dst->bDistributeModeAnalysis = src->bDistributeModeAnalysis;
     dst->bDistributeMotionEstimation = src->bDistributeMotionEstimation;
@@ -2976,6 +3007,9 @@
     dst->bEnableHRDConcatFlag = src->bEnableHRDConcatFlag;
     dst->dolbyProfile = src->dolbyProfile;
     dst->bEnableSvtHevc = src->bEnableSvtHevc;
+    dst->bThreadedME = src->bThreadedME;
+    dst->tmeTaskBlockSize = src->tmeTaskBlockSize;
+    dst->tmeNumBufferRows = src->tmeNumBufferRows;
     dst->bEnableFades = src->bEnableFades;
     dst->bEnableSceneCutAwareQp = src->bEnableSceneCutAwareQp;
     dst->fwdMaxScenecutWindow = src->fwdMaxScenecutWindow;
@@ -3017,11 +3051,9 @@
     memcpy(dst->svtHevcParam, src->svtHevcParam, sizeof(EB_H265_ENC_CONFIGURATION));
 #endif
     /* Film grain */
-    if (src->filmGrain)
-        dst->filmGrain = src->filmGrain;
+    dst->filmGrain = src->filmGrain;
     /* Aom Film grain*/
-    if (src->aomFilmGrain)
-        dst->aomFilmGrain = src->aomFilmGrain;
+    dst->aomFilmGrain = src->aomFilmGrain;
     dst->bEnableSBRC = src->bEnableSBRC;
     dst->bConfigRCFrame = src->bConfigRCFrame;
     dst->isAbrLadderEnable = src->isAbrLadderEnable;

x265-4.1.tar/source/common/picyuv.cpp -> x265-4.2.tar/source/common/picyuv.cpp Changed

x265-4.1.tar/source/common/primitives.cpp -> x265-4.2.tar/source/common/primitives.cpp Changed

x265-4.1.tar/source/common/quant.cpp -> x265-4.2.tar/source/common/quant.cpp Changed

x265-4.2.tar/source/common/riscv64 Added

x265-4.2.tar/source/common/riscv64/asm-primitives.cpp Added

@@ -0,0 +1,495 @@
+/*****************************************************************************
+ * Copyright (C) 2020 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+
+#include "common.h"
+#include "primitives.h"
+#include "x265.h"
+#include "cpu.h"
+
+extern "C" {
+#include "fun-decls.h"
+}
+
+#define ALL_LUMA_TU_TYPED_L(prim, fncdef, fname, cpu) \
+    p.cuBLOCK_4x4.prim   = fncdef PFX(fname ## _2_ ## cpu); \
+    p.cuBLOCK_8x8.prim   = fncdef PFX(fname ## _3_ ## cpu); \
+    p.cuBLOCK_16x16.prim = fncdef PFX(fname ## _4_ ## cpu); \
+    p.cuBLOCK_32x32.prim = fncdef PFX(fname ## _5_ ## cpu);
+#define ALL_LUMA_TU_L(prim, fname, cpu)  ALL_LUMA_TU_TYPED_L(prim, , fname, cpu)
+
+#define ALL_LUMA_TU_TYPED_S(prim, fncdef, fname, cpu) \
+    p.cuBLOCK_4x4.prim   = fncdef PFX(fname ## _4_ ## cpu); \
+    p.cuBLOCK_8x8.prim   = fncdef PFX(fname ## _8_ ## cpu); \
+    p.cuBLOCK_16x16.prim = fncdef PFX(fname ## _16_ ## cpu); \
+    p.cuBLOCK_32x32.prim = fncdef PFX(fname ## _32_ ## cpu);
+#define ALL_LUMA_TU_S(prim, fname, cpu)  ALL_LUMA_TU_TYPED_S(prim, , fname, cpu)
+
+#define ALL_LUMA_BLOCKS_TYPED_S(prim, fncdef, fname, cpu) \
+    p.cuBLOCK_4x4.prim   = fncdef PFX(fname ## _4_ ## cpu); \
+    p.cuBLOCK_8x8.prim   = fncdef PFX(fname ## _8_ ## cpu); \
+    p.cuBLOCK_16x16.prim = fncdef PFX(fname ## _16_ ## cpu); \
+    p.cuBLOCK_32x32.prim = fncdef PFX(fname ## _32_ ## cpu); \
+    p.cuBLOCK_64x64.prim = fncdef PFX(fname ## _64_ ## cpu);
+#define ALL_LUMA_BLOCKS_S(prim, fname, cpu)  ALL_LUMA_BLOCKS_TYPED_S(prim, , fname, cpu)
+
+#define ALL_LUMA_BLOCKS_TYPED(prim, fncdef, fname, cpu) \
+    p.cuBLOCK_4x4.prim   = fncdef PFX(fname ## _4x4_ ## cpu); \
+    p.cuBLOCK_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.cuBLOCK_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.cuBLOCK_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu); \
+    p.cuBLOCK_64x64.prim = fncdef PFX(fname ## _64x64_ ## cpu);
+#define ALL_LUMA_BLOCKS(prim, fname, cpu)  ALL_LUMA_BLOCKS_TYPED(prim, , fname, cpu)
+
+#define ALL_LUMA_BLOCKS_TYPED_B(prim, fncdef, fname, cpu) \
+    p.cuBLOCK_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.cuBLOCK_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.cuBLOCK_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu); \
+    p.cuBLOCK_64x64.prim = fncdef PFX(fname ## _64x64_ ## cpu);
+#define ALL_LUMA_BLOCKS_B(prim, fname, cpu)  ALL_LUMA_BLOCKS_TYPED_B(prim, , fname, cpu)
+
+#define ALL_LUMA_PU_TYPED(prim, fncdef, fname, cpu)                            \
+  p.puLUMA_4x4.prim = fncdef PFX(fname##_4x4_##cpu);                         \
+  p.puLUMA_8x8.prim = fncdef PFX(fname##_8x8_##cpu);                         \
+  p.puLUMA_16x16.prim = fncdef PFX(fname##_16x16_##cpu);                     \
+  p.puLUMA_32x32.prim = fncdef PFX(fname##_32x32_##cpu);                     \
+  p.puLUMA_64x64.prim = fncdef PFX(fname##_64x64_##cpu);                     \
+  p.puLUMA_8x4.prim = fncdef PFX(fname##_8x4_##cpu);                         \
+  p.puLUMA_4x8.prim = fncdef PFX(fname##_4x8_##cpu);                         \
+  p.puLUMA_16x8.prim = fncdef PFX(fname##_16x8_##cpu);                       \
+  p.puLUMA_8x16.prim = fncdef PFX(fname##_8x16_##cpu);                       \
+  p.puLUMA_16x32.prim = fncdef PFX(fname##_16x32_##cpu);                     \
+  p.puLUMA_32x16.prim = fncdef PFX(fname##_32x16_##cpu);                     \
+  p.puLUMA_64x32.prim = fncdef PFX(fname##_64x32_##cpu);                     \
+  p.puLUMA_32x64.prim = fncdef PFX(fname##_32x64_##cpu);                     \
+  p.puLUMA_16x12.prim = fncdef PFX(fname##_16x12_##cpu);                     \
+  p.puLUMA_12x16.prim = fncdef PFX(fname##_12x16_##cpu);                     \
+  p.puLUMA_16x4.prim = fncdef PFX(fname##_16x4_##cpu);                       \
+  p.puLUMA_4x16.prim = fncdef PFX(fname##_4x16_##cpu);                       \
+  p.puLUMA_32x24.prim = fncdef PFX(fname##_32x24_##cpu);                     \
+  p.puLUMA_24x32.prim = fncdef PFX(fname##_24x32_##cpu);                     \
+  p.puLUMA_32x8.prim = fncdef PFX(fname##_32x8_##cpu);                       \
+  p.puLUMA_8x32.prim = fncdef PFX(fname##_8x32_##cpu);                       \
+  p.puLUMA_64x48.prim = fncdef PFX(fname##_64x48_##cpu);                     \
+  p.puLUMA_48x64.prim = fncdef PFX(fname##_48x64_##cpu);                     \
+  p.puLUMA_64x16.prim = fncdef PFX(fname##_64x16_##cpu);                     \
+  p.puLUMA_16x64.prim = fncdef PFX(fname##_16x64_##cpu)
+#define ALL_LUMA_PU(prim, fname, cpu) ALL_LUMA_PU_TYPED(prim, , fname, cpu)
+
+#define ALL_CHROMA_420_CU_TYPED(prim, fncdef, fname, cpu) \
+    p.chromaX265_CSP_I420.cuBLOCK_420_4x4.prim   = fncdef PFX(fname ## _4x4_ ## cpu); \
+    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu)
+#define ALL_CHROMA_420_CU(prim, fname, cpu) ALL_CHROMA_420_CU_TYPED(prim, , fname, cpu)
+
+#define ALL_CHROMA_420_CU_TYPED_B(prim, fncdef, fname, cpu) \
+    p.chromaX265_CSP_I420.cuBLOCK_420_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.chromaX265_CSP_I420.cuBLOCK_420_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.chromaX265_CSP_I420.cuBLOCK_420_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu)
+#define ALL_CHROMA_420_CU_B(prim, fname, cpu) ALL_CHROMA_420_CU_TYPED_B(prim, , fname, cpu)
+
+#define ALL_CHROMA_422_CU_TYPED(prim, fncdef, fname, cpu) \
+    p.chromaX265_CSP_I422.cuBLOCK_422_4x8.prim   = fncdef PFX(fname ## _4x8_ ## cpu); \
+    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.prim  = fncdef PFX(fname ## _8x16_ ## cpu); \
+    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.prim = fncdef PFX(fname ## _16x32_ ## cpu); \
+    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.prim = fncdef PFX(fname ## _32x64_ ## cpu)
+#define ALL_CHROMA_422_CU(prim, fname, cpu) ALL_CHROMA_422_CU_TYPED(prim, , fname, cpu)
+
+#define ALL_CHROMA_422_CU_TYPED_B(prim, fncdef, fname, cpu) \
+    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.prim  = fncdef PFX(fname ## _8x16_ ## cpu); \
+    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.prim = fncdef PFX(fname ## _16x32_ ## cpu); \
+    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.prim = fncdef PFX(fname ## _32x64_ ## cpu)
+#define ALL_CHROMA_422_CU_B(prim, fname, cpu) ALL_CHROMA_422_CU_TYPED_B(prim, , fname, cpu)
+
+#define ALL_CHROMA_420_PU_TYPED(prim, fncdef, fname, cpu)                                  \
+    p.chromaX265_CSP_I420.puCHROMA_420_4x4.prim   = fncdef PFX(fname ## _4x4_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_4x2.prim   = fncdef PFX(fname ## _4x2_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x4.prim   = fncdef PFX(fname ## _8x4_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_4x8.prim   = fncdef PFX(fname ## _4x8_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.prim  = fncdef PFX(fname ## _16x8_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.prim  = fncdef PFX(fname ## _8x16_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.prim = fncdef PFX(fname ## _32x16_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.prim = fncdef PFX(fname ## _16x32_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x6.prim   = fncdef PFX(fname ## _8x6_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_6x8.prim   = fncdef PFX(fname ## _6x8_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x2.prim   = fncdef PFX(fname ## _8x2_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.prim = fncdef PFX(fname ## _16x12_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_12x16.prim = fncdef PFX(fname ## _12x16_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.prim  = fncdef PFX(fname ## _16x4_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_4x16.prim  = fncdef PFX(fname ## _4x16_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.prim = fncdef PFX(fname ## _32x24_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.prim = fncdef PFX(fname ## _24x32_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.prim  = fncdef PFX(fname ## _32x8_ ## cpu); \
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.prim  = fncdef PFX(fname ## _8x32_ ## cpu)
+#define ALL_CHROMA_420_PU(prim, fname, cpu) ALL_CHROMA_420_PU_TYPED(prim, , fname, cpu)
+
+#define ALL_CHROMA_422_PU_TYPED(prim, fncdef, fname, cpu)               \
+    p.chromaX265_CSP_I422.puCHROMA_422_4x8.prim   = fncdef PFX(fname ## _4x8_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.prim  = fncdef PFX(fname ## _8x16_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.prim = fncdef PFX(fname ## _16x32_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.prim = fncdef PFX(fname ## _32x64_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_4x4.prim   = fncdef PFX(fname ## _4x4_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_4x16.prim  = fncdef PFX(fname ## _4x16_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x32.prim  = fncdef PFX(fname ## _8x32_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.prim = fncdef PFX(fname ## _16x64_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x12.prim  = fncdef PFX(fname ## _8x12_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_6x16.prim  = fncdef PFX(fname ## _6x16_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x4.prim   = fncdef PFX(fname ## _8x4_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_16x24.prim = fncdef PFX(fname ## _16x24_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.prim = fncdef PFX(fname ## _12x32_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.prim  = fncdef PFX(fname ## _16x8_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_4x32.prim  = fncdef PFX(fname ## _4x32_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_32x48.prim = fncdef PFX(fname ## _32x48_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_24x64.prim = fncdef PFX(fname ## _24x64_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.prim = fncdef PFX(fname ## _32x16_ ## cpu); \
+    p.chromaX265_CSP_I422.puCHROMA_422_8x64.prim  = fncdef PFX(fname ## _8x64_ ## cpu)
+#define ALL_CHROMA_422_PU(prim, fname, cpu) ALL_CHROMA_422_PU_TYPED(prim, , fname, cpu)
+
+#define ALL_CHROMA_444_PU_TYPED(prim, fncdef, fname, cpu) \
+    p.chromaX265_CSP_I444.puLUMA_4x4.prim   = fncdef PFX(fname ## _4x4_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_8x8.prim   = fncdef PFX(fname ## _8x8_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x16.prim = fncdef PFX(fname ## _16x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_32x32.prim = fncdef PFX(fname ## _32x32_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_64x64.prim = fncdef PFX(fname ## _64x64_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_8x4.prim   = fncdef PFX(fname ## _8x4_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_4x8.prim   = fncdef PFX(fname ## _4x8_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x8.prim  = fncdef PFX(fname ## _16x8_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_8x16.prim  = fncdef PFX(fname ## _8x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x32.prim = fncdef PFX(fname ## _16x32_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_32x16.prim = fncdef PFX(fname ## _32x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_64x32.prim = fncdef PFX(fname ## _64x32_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_32x64.prim = fncdef PFX(fname ## _32x64_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x12.prim = fncdef PFX(fname ## _16x12_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_12x16.prim = fncdef PFX(fname ## _12x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x4.prim  = fncdef PFX(fname ## _16x4_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_4x16.prim  = fncdef PFX(fname ## _4x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_32x24.prim = fncdef PFX(fname ## _32x24_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_24x32.prim = fncdef PFX(fname ## _24x32_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_32x8.prim  = fncdef PFX(fname ## _32x8_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_8x32.prim  = fncdef PFX(fname ## _8x32_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_64x48.prim = fncdef PFX(fname ## _64x48_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_48x64.prim = fncdef PFX(fname ## _48x64_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_64x16.prim = fncdef PFX(fname ## _64x16_ ## cpu); \
+    p.chromaX265_CSP_I444.puLUMA_16x64.prim = fncdef PFX(fname ## _16x64_ ## cpu)
+#define ALL_CHROMA_444_PU(prim, fname, cpu) ALL_CHROMA_444_PU_TYPED(prim, , fname, cpu)
+
+#define LUMA_CU(W, H) \
+    p.cuBLOCK_ ## W ## x ## H.sub_ps        = pixel_sub_ps_c<W, H>;
+
+#if defined(__GNUC__)
+#define GCC_VERSION (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__)
+#endif
+
+#include "fun-decls-prim.h"
+
+namespace X265_NS
+{
+// private x265 namespace
+
+void setupRVVPrimitives(EncoderPrimitives &p)
+{
+    // p2s
+    ALL_CHROMA_420_PU(p2sNONALIGNED, filterPixelToShort, rvv);
+    ALL_CHROMA_422_PU(p2sALIGNED, filterPixelToShort, rvv);
+    ALL_CHROMA_444_PU(p2sALIGNED, filterPixelToShort, rvv);
+    ALL_LUMA_PU(convert_p2sALIGNED, filterPixelToShort, rvv);
+    ALL_CHROMA_420_PU(p2sALIGNED, filterPixelToShort, rvv);
+    ALL_CHROMA_422_PU(p2sNONALIGNED, filterPixelToShort, rvv);
+    ALL_CHROMA_444_PU(p2sNONALIGNED, filterPixelToShort, rvv);
+    ALL_LUMA_PU(convert_p2sNONALIGNED, filterPixelToShort, rvv);
+
+    // copy_count
+    ALL_LUMA_TU_S(copy_cnt, copy_cnt, v);
+    ALL_LUMA_TU_S(count_nonzero, count_nonzero, v);
+
+    p.idst4x4               = PFX(idst4_v);
+    p.denoiseDct            = PFX(denoiseDct_v);
+    p.dst4x4                = PFX(dst4_v);
+
+    ALL_LUMA_TU_S(dct, dct, v);
+    ALL_LUMA_TU_S(idct, idct, v);
+
+    ALL_LUMA_TU_L(nonPsyRdoQuant, nonPsyRdoQuant, v);
+    ALL_LUMA_TU_L(psyRdoQuant, PsyRdoQuant, v);
+    ALL_LUMA_TU_L(psyRdoQuant_1p, nonPsyRdoQuant, v);
+    ALL_LUMA_TU_L(psyRdoQuant_2p, PsyRdoQuant, v);
+
+    // ssd_s
+    ALL_LUMA_BLOCKS(ssd_sNONALIGNED, pixel_ssd_s, rvv);
+    ALL_LUMA_BLOCKS(ssd_sALIGNED, pixel_ssd_s, rvv);
+
+    // sse_pp
+    ALL_LUMA_BLOCKS(sse_pp, pixel_sse_pp, rvv);
+
+    ALL_CHROMA_420_CU(sse_pp, pixel_sse_pp, rvv);
+    ALL_CHROMA_422_CU(sse_pp, pixel_sse_pp, rvv);
+
+    // sse_ss
+    ALL_LUMA_BLOCKS(sse_ss, pixel_sse_ss, rvv);
+
+    // sad
+    ALL_LUMA_PU(sad, pixel_sad, rvv);
+    ALL_LUMA_PU(sad_x3, sad_x3, rvv);
+    ALL_LUMA_PU(sad_x4, sad_x4, rvv);
+
+#if !HIGH_BIT_DEPTH
+    // pixel_avg_pp
+    ALL_LUMA_PU(pixelavg_ppNONALIGNED, pixel_avg_pp, rvv);
+    ALL_LUMA_PU(pixelavg_ppALIGNED, pixel_avg_pp, rvv);
+
+    // intra_pred_planar
+    p.cuBLOCK_4x4.intra_predPLANAR_IDX = PFX(intra_pred_planar4_rvv);
+    p.cuBLOCK_8x8.intra_predPLANAR_IDX = PFX(intra_pred_planar8_rvv);
+    p.cuBLOCK_16x16.intra_predPLANAR_IDX = PFX(intra_pred_planar16_rvv);
+
+    // ssimDist
+    ALL_LUMA_BLOCKS_S(ssimDist, ssimDist, v);
+
+    // normFact
+    p.cuBLOCK_8x8.normFact = PFX(normFact_v);
+    p.cuBLOCK_16x16.normFact = PFX(normFact_v);
+    p.cuBLOCK_32x32.normFact = PFX(normFact_v);
+    p.cuBLOCK_64x64.normFact = PFX(normFact_v);
+
+    p.costCoeffNxN = PFX(costCoeffNxN_rvv);
+
+    p.cuBLOCK_4x4.sa8d   = PFX(satd4_4x4_rvv);
+    ALL_LUMA_BLOCKS_B(sa8d, sa8d, rvv);
+
+    p.chromaX265_CSP_I420.cuBLOCK_16x16.sa8d = PFX(sa8d_8x8_rvv);
+    p.chromaX265_CSP_I420.cuBLOCK_32x32.sa8d = PFX(sa8d_16x16_rvv);
+    p.chromaX265_CSP_I420.cuBLOCK_64x64.sa8d = PFX(sa8d_32x32_rvv);
+
+    p.chromaX265_CSP_I422.cuBLOCK_16x16.sa8d = PFX(sa8d_8x16_rvv);
+    p.chromaX265_CSP_I422.cuBLOCK_32x32.sa8d = PFX(sa8d_16x32_rvv);
+    p.chromaX265_CSP_I422.cuBLOCK_64x64.sa8d = PFX(sa8d_32x64_rvv);
+
+    ALL_CHROMA_422_CU_B(sa8d, sa8d, rvv);
+
+    p.puLUMA_4x4.satd     = PFX(satd4_4x4_rvv);
+    p.puLUMA_4x8.satd     = PFX(satd4_4x8_rvv);
+    p.puLUMA_4x16.satd    = PFX(satd4_4x16_rvv);
+    p.puLUMA_12x16.satd   = PFX(satd4_12x16_rvv);
+    p.puLUMA_8x4.satd     = PFX(satd8_8x4_rvv);
+    p.puLUMA_8x8.satd     = PFX(satd8_8x8_rvv);
+    p.puLUMA_8x16.satd    = PFX(satd8_8x16_rvv);
+    p.puLUMA_8x32.satd    = PFX(satd8_8x32_rvv);
+    p.puLUMA_16x4.satd    = PFX(satd8_16x4_rvv);
+    p.puLUMA_16x8.satd    = PFX(satd8_16x8_rvv);
+    p.puLUMA_16x12.satd   = PFX(satd8_16x12_rvv);
+    p.puLUMA_16x16.satd   = PFX(satd8_16x16_rvv);
+    p.puLUMA_16x32.satd   = PFX(satd8_16x32_rvv);
+    p.puLUMA_16x64.satd   = PFX(satd8_16x64_rvv);
+    p.puLUMA_24x32.satd   = PFX(satd8_24x32_rvv);
+    p.puLUMA_32x8.satd    = PFX(satd8_32x8_rvv);
+    p.puLUMA_32x16.satd   = PFX(satd8_32x16_rvv);
+    p.puLUMA_32x24.satd   = PFX(satd8_32x24_rvv);
+    p.puLUMA_32x32.satd   = PFX(satd8_32x32_rvv);
+    p.puLUMA_32x64.satd   = PFX(satd8_32x64_rvv);
+    p.puLUMA_48x64.satd   = PFX(satd8_48x64_rvv);
+    p.puLUMA_64x16.satd   = PFX(satd8_64x16_rvv);
+    p.puLUMA_64x32.satd   = PFX(satd8_64x32_rvv);
+    p.puLUMA_64x48.satd   = PFX(satd8_64x48_rvv);
+    p.puLUMA_64x64.satd   = PFX(satd8_64x64_rvv);
+
+    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd    = PFX(satd4_4x4_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_4x8.satd    = PFX(satd4_4x8_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_4x16.satd   = PFX(satd4_4x16_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_12x16.satd  = PFX(satd4_12x16_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd    = PFX(satd8_8x4_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_8x8.satd    = PFX(satd8_8x8_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.satd   = PFX(satd8_8x16_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.satd   = PFX(satd8_8x32_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.satd   = PFX(satd8_16x4_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.satd   = PFX(satd8_16x8_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.satd  = PFX(satd8_16x12_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.satd  = PFX(satd8_16x16_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.satd  = PFX(satd8_16x32_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.satd  = PFX(satd8_24x32_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.satd   = PFX(satd8_32x8_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.satd  = PFX(satd8_32x16_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.satd  = PFX(satd8_32x24_rvv);
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.satd  = PFX(satd8_32x32_rvv);
+
+    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd   = PFX(satd4_4x4_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd   = PFX(satd4_4x8_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_4x16.satd   = PFX(satd4_4x16_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd   = PFX(satd4_4x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd   = PFX(satd4_12x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd  = PFX(satd8_8x4_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x8.satd  = PFX(satd8_8x8_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd  = PFX(satd8_8x12_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.satd  = PFX(satd8_8x16_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x32.satd  = PFX(satd8_8x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_8x64.satd  = PFX(satd8_8x64_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd  = PFX(satd4_4x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd = PFX(satd4_12x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.satd  = PFX(satd8_16x8_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.satd  = PFX(satd8_16x16_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_16x24.satd  = PFX(satd8_16x24_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.satd  = PFX(satd8_16x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.satd  = PFX(satd8_16x64_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_24x64.satd  = PFX(satd8_24x64_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.satd  = PFX(satd8_32x16_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.satd  = PFX(satd8_32x32_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_32x48.satd  = PFX(satd8_32x48_rvv);
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.satd  = PFX(satd8_32x64_rvv);
+#endif
+
+    // pixel_var
+    ALL_LUMA_BLOCKS_B(var, pixel_var, v);
+
+    // calc_Residual
+    ALL_LUMA_TU_S(calcresidualNONALIGNED, getResidual, v);
+    ALL_LUMA_TU_S(calcresidualALIGNED, getResidual, v);
+
+    // pixel_sub_ps
+    ALL_LUMA_BLOCKS(sub_ps, pixel_sub_ps, v);
+    ALL_CHROMA_420_CU(sub_ps, pixel_sub_ps, v);
+    ALL_CHROMA_422_CU(sub_ps, pixel_sub_ps, v);
+
+    // pixel_add_ps
+    ALL_LUMA_BLOCKS(add_psNONALIGNED, pixel_add_ps, v);
+    ALL_LUMA_BLOCKS(add_psALIGNED, pixel_add_ps, v);
+    ALL_CHROMA_420_CU(add_psNONALIGNED, pixel_add_ps, v);
+    ALL_CHROMA_420_CU(add_psALIGNED, pixel_add_ps, v);
+    ALL_CHROMA_422_CU(add_psNONALIGNED, pixel_add_ps, v);
+    ALL_CHROMA_422_CU(add_psALIGNED, pixel_add_ps, v);
+
+    //scale2D_64to32
+    p.scale2D_64to32  = PFX(scale2D_64to32_v);
+
+    // scale1D_128to64
+    p.scale1D_128to64NONALIGNED = PFX(scale1D_128to64_v);
+    p.scale1D_128to64ALIGNED = PFX(scale1D_128to64_v);
+
+    // quant
+    p.dequant_scaling = PFX(dequant_scaling_v);
+    p.dequant_normal = PFX(dequant_normal_v);
+    p.quant = PFX(quant_v);
+    p.nquant = PFX(nquant_v);
+
+    // ssim_4x4x2_core
+    p.ssim_4x4x2_core = PFX(ssim_4x4x2_core_v);
+
+    p.scanPosLast = PFX(scanPosLast_v);
+
+    p.saoCuStatsE0 = PFX(saoCuStatsE0_rvv);
+    p.saoCuStatsE1 = PFX(saoCuStatsE1_rvv);
+    p.saoCuStatsE2 = PFX(saoCuStatsE2_rvv);
+    p.saoCuStatsE3 = PFX(saoCuStatsE3_rvv);
+
+    p.saoCuOrgE0 = PFX(processSaoCUE0_rvv);
+    p.saoCuOrgE1 = PFX(processSaoCUE1_rvv);
+    p.saoCuOrgE1_2Rows = PFX(processSaoCUE1_2Rows_rvv);
+    p.saoCuOrgE20 = PFX(processSaoCUE2_rvv);
+    p.saoCuOrgE21 = PFX(processSaoCUE2_rvv);
+    p.saoCuOrgE30 = PFX(processSaoCUE3_rvv);
+    p.saoCuOrgE31 = PFX(processSaoCUE3_rvv);
+    p.saoCuOrgB0 = PFX(processSaoCUB0_rvv);
+    p.sign = PFX(calSign_rvv);
+    p.pelFilterLumaStrong0 = PFX(pelFilterLumaStrong_v_rvv);
+    p.pelFilterLumaStrong1 = PFX(pelFilterLumaStrong_h_rvv);
+    //p.pelFilterChroma0     = PFX(pelFilterChroma_V_rvv);
+    //p.pelFilterChroma1     = PFX(pelFilterChroma_H_rvv);
+
+    p.weight_pp = PFX(weight_pp_v);
+    p.weight_sp = PFX(weight_sp_v);
+
+    p.planecopy_cp = PFX(planecopy_cp_v);
+    p.planecopy_sp = PFX(planecopy_sp_v);
+    p.planecopy_sp_shl = PFX(planecopy_sp_shl_v);
+    p.planecopy_pp_shr = PFX(planecopy_pp_shr_v);
+
+    ALL_LUMA_PU(addAvgNONALIGNED, addAvg, v);
+    ALL_LUMA_PU(addAvgALIGNED, addAvg, v);
+    ALL_CHROMA_420_PU(addAvgNONALIGNED, addAvg, v);
+    ALL_CHROMA_420_PU(addAvgALIGNED, addAvg, v);
+    ALL_CHROMA_422_PU(addAvgNONALIGNED, addAvg, v);
+    ALL_CHROMA_422_PU(addAvgALIGNED, addAvg, v);
+
+    ALL_LUMA_PU(copy_pp, blockcopy_pp, v);
+    ALL_CHROMA_420_PU(copy_pp, blockcopy_pp, v);
+    ALL_CHROMA_422_PU(copy_pp, blockcopy_pp, v);
+
+    ALL_LUMA_BLOCKS(copy_ss, blockcopy_ss, v);
+    ALL_CHROMA_420_CU(copy_ss, blockcopy_ss, v);
+    ALL_CHROMA_422_CU(copy_ss, blockcopy_ss, v);
+
+    ALL_LUMA_BLOCKS(copy_sp, blockcopy_sp, v);
+    ALL_CHROMA_420_CU(copy_sp, blockcopy_sp, v);
+    ALL_CHROMA_422_CU(copy_sp, blockcopy_sp, v);
+
+    ALL_LUMA_BLOCKS(copy_ps, blockcopy_ps, v);
+    ALL_CHROMA_420_CU(copy_ps, blockcopy_ps, v);
+    ALL_CHROMA_422_CU(copy_ps, blockcopy_ps, v);
+
+    ALL_LUMA_BLOCKS_S(blockfill_sNONALIGNED, blockfill_s, v);
+    ALL_LUMA_BLOCKS_S(blockfill_sALIGNED, blockfill_s, v);
+    ALL_LUMA_BLOCKS_S(cpy2Dto1D_shl, cpy2Dto1D_shl, v);
+    ALL_LUMA_BLOCKS_S(cpy2Dto1D_shr, cpy2Dto1D_shr, v);
+    ALL_LUMA_BLOCKS_S(cpy1Dto2D_shlNONALIGNED, cpy1Dto2D_shl, v);
+    ALL_LUMA_BLOCKS_S(cpy1Dto2D_shlALIGNED, cpy1Dto2D_shl, v);
+    ALL_LUMA_BLOCKS_S(cpy1Dto2D_shr, cpy1Dto2D_shr, v);
+}
+
+void setupAssemblyPrimitives(EncoderPrimitives &p, int cpuMask)
+{
+    if (cpuMask & X265_CPU_RVV)
+    {
+        setupRVVPrimitives(p);
+
+        if (cpuMask & X265_CPU_ZBB) {
+#if !HIGH_BIT_DEPTH
+            ALL_LUMA_BLOCKS(psy_cost_pp, psyCost_pp, rvv);
+#endif
+        }
+    }
+}
+
+void setupIntrinsicPrimitives(EncoderPrimitives &p, int cpuMask)
+{
+#if HAVE_RVV_INTRINSIC
+    if (cpuMask & X265_CPU_RVV)
+    {
+        setupPixelPrimitives_rvv(p);
+        setupSaoPrimitives_rvv(p);
+        setupIntraPrimitives_rvv(p);
+#if !HIGH_BIT_DEPTH
+        setupFilterPrimitives_rvv(p);
+#endif
+    }
+#else
+    (void)p;
+    (void)cpuMask;
+#endif
+}
+
+} // namespace X265_NS

x265-4.2.tar/source/common/riscv64/asm.S Added

@@ -0,0 +1,340 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef ASM_S_  // #include guards
+#define ASM_S_
+
+.option arch, +v
+
+#define PFX3(prefix, name) prefix ## _ ## name
+#define PFX2(prefix, name) PFX3(prefix, name)
+#define PFX(name)          PFX2(X265_NS, name)
+
+#if defined __clang__
+#define EXTERN_ASM
+#define HAVE_AS_FUNC 0
+#define PREFIX 1
+#else
+#define EXTERN_ASM
+#define HAVE_AS_FUNC 1
+#endif
+
+#ifdef __ELF__
+#define ELF
+#else
+#ifdef PREFIX
+#define ELF #
+#else
+#define ELF @
+#endif
+#endif
+
+#if HAVE_AS_FUNC
+#define FUNC
+#else
+#ifdef PREFIX
+#define FUNC #
+#else
+#define FUNC @
+#endif
+#endif
+
+#define GLUE(a, b) a ## b
+#define JOIN(a, b) GLUE(a, b)
+
+#define PFX_C(name)        JOIN(JOIN(JOIN(EXTERN_ASM, X265_NS), _), name)
+
+#define FENC_STRIDE 64
+#define FDNC_STRIDE 128
+
+// Alignment of stack arguments of size less than 8 bytes.
+#define STACK_ARG_ALIGNMENT 8
+
+// Get offset from SP of stack argument at index `idx`.
+#define STACK_ARG_OFFSET(idx) (idx * STACK_ARG_ALIGNMENT)
+
+.macro function name, export=1
+    .macro endfunc
+ELF     .size   \name, . - \name
+FUNC    .endfunc
+        .purgem endfunc
+    .endm
+        .align  2
+.if \export == 1
+        .global EXTERN_ASM\name
+ELF     .hidden EXTERN_ASM\name
+ELF     .type   EXTERN_ASM\name, %function
+FUNC    .func   EXTERN_ASM\name
+EXTERN_ASM\name:
+.else
+ELF     .hidden \name
+ELF     .type   \name, %function
+FUNC    .func   \name
+\name:
+.endif
+.endm
+
+.macro  const   name, align=2
+    .macro endconst
+ELF     .size   \name, . - \name
+        .purgem endconst
+    .endm
+#ifdef __MACH__
+    .const_data
+#else
+    .section .rodata
+#endif
+    .align          \align
+\name:
+.endm
+
+.macro SUMSUB_AB sum, diff, a, b
+    vadd.vv         \sum,  \a, \b
+    vsub.vv         \diff, \a, \b
+.endm
+
+.macro SUMSUB_ABCD s1, d1, s2, d2, a, b, c, d
+    SUMSUB_AB       \s1, \d1, \a, \b
+    SUMSUB_AB       \s2, \d2, \c, \d
+.endm
+
+.macro HADAMARD4 r1, r2, r3, r4, t1, t2, t3, t4
+    SUMSUB_ABCD     \t1, \t2, \t3, \t4, \r1, \r2, \r3, \r4
+    SUMSUB_ABCD     \r1, \r3, \r2, \r4, \t1, \t3, \t2, \t4
+.endm
+
+.macro SIGNOF x, t1, t2
+    srai            \t1, \x, 31
+    neg             \x, \x
+    li              \t2, 63
+    srl             \x, \x, \t2
+    or              \x, \x, \t1
+.endm
+
+.macro SIGNOF_RVV  vd, va, vb
+    vmv.v.i         \vd, 0
+    vmsltu.vv       v0, \va, \vb
+    vmerge.vim      \vd, \vd, -1, v0
+    vmsgtu.vv       v0, \va, \vb
+    vmerge.vim      \vd, \vd, 1, v0
+.endm
+
+.macro VABS vd, va, vb
+#if HAVE_RVV_ZF
+    vabs.v          \vd, \va
+#else
+    vrsub.vi        \vb, \va, 0
+    vmax.vv         \vd, \va, \vb
+#endif
+.endm
+
+.macro VABD d0, s0, s1, t0
+#if HAVE_RVV_ZF
+    vabd.vv         \d0, \s0, \s1
+#else
+    vmax.vv         \d0, \s0, \s1
+    vmin.vv         \t0, \s0, \s1
+    vsub.vv         \d0, \d0, \t0
+#endif
+.endm
+
+.macro VABDU d0, s0, s1, t0
+#if HAVE_RVV_ZF
+    vabdu.vv        \d0, \s0, \s1
+#else
+    vmaxu.vv        \d0, \s0, \s1
+    vminu.vv        \t0, \s0, \s1
+    vsub.vv         \d0, \d0, \t0
+#endif
+.endm
+
+.macro UWABDU d0, s0, s1, t0, t1
+    vmaxu.vv        \t1, \s0, \s1
+    vminu.vv        \t0, \s0, \s1
+    vwsubu.vv       \d0, \t1, \t0
+.endm
+
+.macro UWADBDACCU d0, s0, s1, t0, t1
+#if HAVE_RVV_ZF
+    vwabdaccu.vv    \d0, \s0, \s1
+#else
+    vmaxu.vv        \t1, \s0, \s1
+    vminu.vv        \t0, \s0, \s1
+    vsub.vv         \t1, \t1, \t0
+    vwaddu.wv       \d0, \d0, \t1
+#endif
+.endm
+
+// v0 must be 0x5 as mask
+.macro TRANSPOSE_4x4 sew, d0, d1, d2, d3, r0, r1, r2, r3
+
+    # Stage 1: Interleave within pairs (\es, VL=4)
+    vslideup.vi     \d0, \r1, 1
+    vslidedown.vi   \d1, \r0, 1
+    vslideup.vi     \d2, \r3, 1
+    vslidedown.vi   \d3, \r2, 1
+
+    # Reinterpret v0 as \es mask
+    vmerge.vvm      \d0, \d0, \r0, v0
+    vmerge.vvm      \d1, \r1, \d1, v0
+    vmerge.vvm      \d2, \d2, \r2, v0
+    vmerge.vvm      \d3, \r3, \d3, v0
+
+    # Stage 2: Switch to \ext_es, VL=2
+.if \sew == 8
+    vsetivli        zero, 2, e16, mf2, ta, ma
+.elseif \sew == 16
+    vsetivli        zero, 2, e32, mf2, ta, ma
+.endif
+
+    vslideup.vi     \r0, \d2, 1
+    vslidedown.vi   \r1, \d0, 1
+    vslideup.vi     \r2, \d3, 1
+    vslidedown.vi   \r3, \d1, 1
+
+    vmerge.vvm      \d0, \r0, \d0, v0
+    vmerge.vvm      \d2, \d2, \r1, v0
+    vmerge.vvm      \d1, \r2, \d1, v0
+    vmerge.vvm      \d3, \d3, \r3, v0
+
+    # Caller should reinterpret as \es and store
+.endm
+
+// v0 must be 0x55 as mask
+.macro TRANSPOSE_8x8 es, d0, d1, d2, d3, d4, d5, d6, d7, r0, r1, r2, r3, r4, r5, r6, r7
+    vslideup.vi     \d0, \r1, 1
+    vslidedown.vi   \d1, \r0, 1
+    vslideup.vi     \d2, \r3, 1
+    vslidedown.vi   \d3, \r2, 1
+    vslideup.vi     \d4, \r5, 1
+    vslidedown.vi   \d5, \r4, 1
+    vslideup.vi     \d6, \r7, 1
+    vslidedown.vi   \d7, \r6, 1
+    vmerge.vvm      \d0, \d0, \r0, v0
+    vmerge.vvm      \d1, \r1, \d1, v0
+    vmerge.vvm      \d2, \d2, \r2, v0
+    vmerge.vvm      \d3, \r3, \d3, v0
+    vmerge.vvm      \d4, \d4, \r4, v0
+    vmerge.vvm      \d5, \r5, \d5, v0
+    vmerge.vvm      \d6, \d6, \r6, v0
+    vmerge.vvm      \d7, \r7, \d7, v0
+
+.if \es == 16
+    vsetivli        zero, 4, e32, m1, ta, ma
+.elseif \es == 8
+    vsetivli        zero, 4, e16, m1, ta, ma
+.endif
+
+    vslideup.vi     \r0, \d2, 1
+    vslidedown.vi   \r1, \d0, 1
+    vslideup.vi     \r2, \d3, 1
+    vslidedown.vi   \r3, \d1, 1
+    vslideup.vi     \r4, \d6, 1
+    vslidedown.vi   \r5, \d4, 1
+    vslideup.vi     \r6, \d7, 1
+    vslidedown.vi   \r7, \d5, 1
+    vmerge.vvm      \r0, \r0, \d0, v0
+    vmerge.vvm      \r1, \d2, \r1, v0
+    vmerge.vvm      \r2, \r2, \d1, v0
+    vmerge.vvm      \r3, \d3, \r3, v0
+    vmerge.vvm      \r4, \r4, \d4, v0
+    vmerge.vvm      \r5, \d6, \r5, v0
+    vmerge.vvm      \r6, \r6, \d5, v0
+    vmerge.vvm      \r7, \d7, \r7, v0
+
+.if \es == 16
+    vsetivli        zero, 2, e64, m1, ta, ma
+.elseif \es == 8
+    vsetivli        zero, 2, e32, m1, ta, ma
+.endif
+
+    vslideup.vi     \d0, \r4, 1
+    vslidedown.vi   \d4, \r0, 1
+    vslideup.vi     \d1, \r6, 1
+    vslidedown.vi   \d5, \r2, 1
+    vslideup.vi     \d2, \r5, 1
+    vslidedown.vi   \d6, \r1, 1
+    vslideup.vi     \d3, \r7, 1
+    vslidedown.vi   \d7, \r3, 1
+    vmerge.vvm      \d0, \d0, \r0, v0
+    vmerge.vvm      \d4, \r4, \d4, v0
+    vmerge.vvm      \d1, \d1, \r2, v0
+    vmerge.vvm      \d5, \r6, \d5, v0
+    vmerge.vvm      \d2, \d2, \r1, v0
+    vmerge.vvm      \d6, \r5, \d6, v0
+    vmerge.vvm      \d3, \d3, \r3, v0
+    vmerge.vvm      \d7, \r7, \d7, v0
+.endm
+
+// v0 = 0x5555
+.macro TRN_2REG d1, d2, vn, vm
+    vslideup.vi     \d1, \vm, 1
+    vslidedown.vi   \d2, \vn, 1
+    vmerge.vvm      \d1, \d1, \vn, v0
+    vmerge.vvm      \d2, \vm, \d2, v0
+.endm
+
+// v0 = 0x5555
+.macro TRN_4REG d1, d2, d3, d4, s1, s2, s3, s4
+    vslideup.vi     \d1, \s2, 1
+    vslideup.vi     \d3, \s4, 1
+    vslidedown.vi   \d2, \s1, 1
+    vslidedown.vi   \d4, \s3, 1
+    vmerge.vvm      \d1, \d1, \s1, v0
+    vmerge.vvm      \d2, \s2, \d2, v0
+    vmerge.vvm      \d3, \d3, \s3, v0
+    vmerge.vvm      \d4, \s4, \d4, v0
+.endm
+
+// v0 = 0x5555
+.macro TRN_8REG d1, d2, d3, d4, d5, d6, d7, d8, s1, s2, s3, s4, s5, s6, s7, s8
+    vslideup.vi     \d1, \s2, 1
+    vslideup.vi     \d3, \s4, 1
+    vslideup.vi     \d5, \s6, 1
+    vslideup.vi     \d7, \s8, 1
+    vslidedown.vi   \d2, \s1, 1
+    vslidedown.vi   \d4, \s3, 1
+    vslidedown.vi   \d6, \s5, 1
+    vslidedown.vi   \d8, \s7, 1
+    vmerge.vvm      \d1, \d1, \s1, v0
+    vmerge.vvm      \d2, \s2, \d2, v0
+    vmerge.vvm      \d3, \d3, \s3, v0
+    vmerge.vvm      \d4, \s4, \d4, v0
+    vmerge.vvm      \d5, \d5, \s5, v0
+    vmerge.vvm      \d6, \s6, \d6, v0
+    vmerge.vvm      \d7, \d7, \s7, v0
+    vmerge.vvm      \d8, \s8, \d8, v0
+.endm
+
+.macro STRIDE_SHIFT t0, t1, stride0, stride1, shift0, shift1
+    slli     \t0, \stride0, \shift0
+    slli     \t1, \stride1, \shift1
+.endm
+
+.macro ADDPTR t0, t1, s0, s1, stride0, stride1, shift0, shift1
+    stride_shift \t0, \t1, \stride0, \stride1, \shift0, \shift1
+    add      \s0, \s0, \t0
+    add      \s1, \s1, \t1
+.endm
+
+#endif

x265-4.2.tar/source/common/riscv64/blockcopy8.S Added

@@ -0,0 +1,100 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+.align 4
+
+.text
+
+// uint32_t copy_count(int16_t* coeff, const int16_t* residual, intptr_t resiStride)
+.macro COPY_CNT_N_V tr, lmul
+function PFX(copy_cnt_\tr\()_v)
+    slli            a2, a2, 1
+    li              a3, \tr
+    vsetvli         zero, a3, e16, m\lmul, ta, ma
+    vmv.v.i         v20, 0
+    vmv.v.i         v24, 1
+    vmv.v.i         v28, 0
+
+.rept \tr / 2
+    vle16.v         v4, (a1)
+    add             a1, a1, a2
+    vle16.v         v8, (a1)
+    add             a1, a1, a2
+    vse16.v         v4, (a0)
+    addi            a0, a0, \tr * 2
+    vse16.v         v8, (a0)
+    addi            a0, a0, \tr * 2
+
+    vmsne.vi        v0, v4, 0
+    vsetvli         zero, a3, e16, m\lmul, ta, mu
+    vadd.vv         v20, v20, v24, v0.t
+    vmsne.vi        v0, v8, 0
+    vadd.vv         v20, v20, v24, v0.t
+    vsetvli         zero, a3, e16, m\lmul, ta, ma
+.endr
+
+    vredsum.vs      v20, v20, v28
+    vmv.x.s         a0, v20
+    ret
+endfunc
+.endm
+
+COPY_CNT_N_V 4, 1
+COPY_CNT_N_V 8, 1
+COPY_CNT_N_V 16, 2
+COPY_CNT_N_V 32, 4
+
+// int  count_nonzero_c(const int16_t* quantCoeff)
+.macro COUNT_NONZERO_N_V tr, lmul
+function PFX(count_nonzero_\tr\()_v)
+    li              a3, \tr
+    vsetvli         zero, a3, e16, m\lmul, ta, ma
+    vmv.v.i         v12, 0
+    vmv.v.i         v16, 1
+    vmv.v.i         v20, 0
+
+.rept \tr / 2
+    vle16.v         v4, (a0)
+    addi            a0, a0, \tr * 2
+    vle16.v         v8, (a0)
+    addi            a0, a0, \tr * 2
+    vmsne.vi        v0, v4, 0
+    vsetvli         zero, a3, e16, m\lmul, ta, mu
+    vadd.vv         v12, v12, v16, v0.t
+    vmsne.vi        v0, v8, 0
+    vadd.vv         v12, v12, v16, v0.t
+    vsetvli         zero, a3, e16, m\lmul, ta, ma
+.endr
+
+    vredsum.vs      v12, v12, v20
+    vmv.x.s         a0, v12
+    ret
+endfunc
+.endm
+
+COUNT_NONZERO_N_V 4, 1
+COUNT_NONZERO_N_V 8, 1
+COUNT_NONZERO_N_V 16, 2
+COUNT_NONZERO_N_V 32, 4

x265-4.2.tar/source/common/riscv64/cpu.h Added

@@ -0,0 +1,94 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef X265_COMMON_RISCV64_CPU_H
+#define X265_COMMON_RISCV64_CPU_H
+
+#include "x265.h"
+
+#if RISCV64_RUNTIME_CPU_DETECT
+
+#if defined(__linux__)
+
+#include <sys/auxv.h>
+
+#define HWCAP_RISCV64_RVV     (1 << ('V' - 'A'))
+
+static int parse_proc_cpuinfo(const char *flag) {
+    FILE *file = fopen("/proc/cpuinfo", "r");
+    if (file == NULL)
+        return 0;
+
+    char line1024;
+    int found = 0;
+
+    while (fgets(line, sizeof(line), file)) {
+        if (strstr(line, flag) != NULL) {
+            found = 1;
+            break;
+        }
+    }
+
+    fclose(file);
+    return found;
+}
+
+static inline uint32_t riscv64_cpu_detect()
+{
+    uint32_t flags = 0;
+
+    unsigned long hwcap = getauxval(AT_HWCAP);
+
+    if (hwcap & HWCAP_RISCV64_RVV) {
+        flags |= X265_CPU_RVV;
+
+        if (parse_proc_cpuinfo("zbb"))
+            flags |= X265_CPU_ZBB;
+    }
+
+    return flags;
+}
+
+#else // defined(__linux__)
+#error                                                                 \
+    "Run-time CPU feature detection selected, but no detection method" \
+    "available for your platform. Rerun cmake configure with"          \
+    "-DRISCV64_RUNTIME_CPU_DETECT=OFF."
+#endif // defined(__linux__)
+
+#else // if AARCH64_RUNTIME_CPU_DETECT
+
+static inline uint32_t riscv64_cpu_detect()
+{
+    uint32_t flags = 0;
+
+#if HAVE_RVV
+    flags |= X265_CPU_RVV;
+    flags |= X265_CPU_ZBB;
+#endif
+    return flags;
+}
+
+#endif // if RISCV64_RUNTIME_CPU_DETECT
+
+#endif // ifndef X265_COMMON_RISCV64_CPU_H

x265-4.2.tar/source/common/riscv64/dct.S Added

@@ -0,0 +1,1538 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+.text
+
+.macro IDCT_MUL_1V4IMM m1, m2, m3, m4, src, d1, d2, d3, d4
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    vmul.vx         \d1, \src, t1
+    vmul.vx         \d2, \src, t2
+    vmul.vx         \d3, \src, t3
+    vmul.vx         \d4, \src, t4
+.endm
+
+.macro IDCT_MULADD_1V4IMM m1, m2, m3, m4, src, d1, d2, d3, d4
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    vmacc.vx        \d1, t1, \src
+    vmacc.vx        \d2, t2, \src
+    vmacc.vx        \d3, t3, \src
+    vmacc.vx        \d4, t4, \src
+.endm
+
+.macro IDCT_MULADD_2x2V2IMM m1, m2, m3, m4, src1, src2, dst1, dst2
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    vmul.vx         \dst1, \src1, t1
+    vmacc.vx        \dst1, t2, \src2
+    vmul.vx         \dst2, \src1, t3
+    vmacc.vx        \dst2, t4, \src2
+.endm
+
+.macro IDCT_SUMSUB_SHIFT dst1, dst2, src1, src2, shift
+    vadd.vv \dst1, \src1, \src2
+    vsub.vv \dst2, \src1, \src2
+    vsll.vi \dst1, \dst1, \shift
+    vsll.vi \dst2, \dst2, \shift
+.endm
+
+.macro TEMP_STORE len, r1, r2, r3, r4
+    addi            sp, sp, -4 * \len
+    addi            t1, sp, \len
+    addi            t2, sp, \len * 2
+    addi            t3, sp, \len * 3
+    addi            a4, a4, 4 * \len
+    vse32.v         \r1, (sp)
+    vse32.v         \r2, (t1)
+    vse32.v         \r3, (t2)
+    vse32.v         \r4, (t3)
+.endm
+
+.macro TEMP_LOAD len, num, r1, r2, r3, r4
+    addi            t1, sp, \num * \len
+    addi            t2, sp, (\num + 1) * \len
+    addi            t3, sp, (\num + 2) * \len
+    addi            t4, sp, (\num + 3) * \len
+    vle32.v         \r1, (t1)
+    vle32.v         \r2, (t2)
+    vle32.v         \r3, (t3)
+    vle32.v         \r4, (t4)
+.endm
+
+.macro PBFI32_STORE8REG dst, off1, off2, shift, strided
+    vnclip.wi       v17, v16, \shift
+    vnclip.wi       v19, v18, \shift
+    vnclip.wi       v21, v20, \shift
+    vnclip.wi       v23, v22, \shift
+
+    addi            t0, \dst, \off1
+    slli            t2, \strided, 1
+    add             t3, t2, \strided
+    add             t1, t0, \strided
+    add             t2, t2, t0
+    add             t3, t3, t0
+
+    vmv.v.i         v0, 5
+    TRANSPOSE_4x4   16, v16, v18, v20, v22, v17, v19, v21, v23
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    vse16.v         v16, (t0)
+    vse16.v         v18, (t1)
+    vse16.v         v20, (t2)
+    vse16.v         v22, (t3)
+
+    vnclip.wi       v17, v24, \shift
+    vnclip.wi       v19, v26, \shift
+    vnclip.wi       v21, v28, \shift
+    vnclip.wi       v23, v30, \shift
+
+    addi            t0, \dst, \off2
+    slli            t2, \strided, 1
+    add             t3, t2, \strided
+    add             t1, t0, \strided
+    add             t2, t2, t0
+    add             t3, t3, t0
+
+    TRANSPOSE_4x4   16, v16, v18, v20, v22, v17, v19, v21, v23
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    vse16.v         v16, (t0)
+    vse16.v         v18, (t1)
+    vse16.v         v20, (t2)
+    vse16.v         v22, (t3)
+.endm
+
+.macro PBFIM_STORE8REG4 dst, off1, shift, strided
+    addi            t0, \dst, \off1
+    li              t1, 0x55
+    vmv.v.x         v0, t1
+    vnclip.wi       v1, v16, \shift
+    vnclip.wi       v2, v18, \shift
+    vnclip.wi       v3, v20, \shift
+    vnclip.wi       v4, v22, \shift
+    vnclip.wi       v5, v24, \shift
+    vnclip.wi       v6, v26, \shift
+    vnclip.wi       v7, v28, \shift
+    vnclip.wi       v8, v30, \shift
+
+    TRANSPOSE_4x4   16, v16, v17, v18, v19, v1, v2, v3, v4
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    TRANSPOSE_4x4   16, v20, v21, v22, v23, v5, v6, v7, v8
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vslideup.vi     v16, v20, 4
+    vslideup.vi     v17, v21, 4
+    vslideup.vi     v18, v22, 4
+    vslideup.vi     v19, v23, 4
+
+    slli            t2, \strided, 1
+    add             t1, t0, \strided
+    add             t2, t2, t0
+    add             t3, t2, \strided
+    vse16.v         v16, (t0)
+    vse16.v         v17, (t1)
+    vse16.v         v18, (t2)
+    vse16.v         v19, (t3)
+.endm
+
+.macro PBFIM_STORE8REG8 dst, off1, shift, strided
+    addi            t0, \dst, \off1
+    li              t1, 0x55
+    vmv.v.x         v0, t1
+    vnclip.wi       v1, v16, \shift
+    vnclip.wi       v2, v18, \shift
+    vnclip.wi       v3, v20, \shift
+    vnclip.wi       v4, v22, \shift
+    vnclip.wi       v5, v24, \shift
+    vnclip.wi       v6, v26, \shift
+    vnclip.wi       v7, v28, \shift
+    vnclip.wi       v8, v30, \shift
+
+    TRANSPOSE_8x8   16, v16, v17, v18, v19, v20, v21, v22, v23, v1, v2, v3, v4, v5, v6, v7, v8
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    slli            t4, \strided, 2
+    slli            t2, \strided, 1
+    add             t1, t0, \strided
+    add             t2, t2, t0
+    add             t3, t2, \strided
+    vse16.v         v16, (t0)
+    vse16.v         v17, (t1)
+    vse16.v         v18, (t2)
+    vse16.v         v19, (t3)
+
+    add             t0, t0, t4
+    add             t1, t1, t4
+    add             t2, t2, t4
+    add             t3, t3, t4
+    vse16.v         v20, (t0)
+    vse16.v         v21, (t1)
+    vse16.v         v22, (t2)
+    vse16.v         v23, (t3)
+.endm
+
+.macro IDCT32_LOAD_LINE strided, src, l1, l2, l3, l4, d1, d2, d3, d4
+    addi            t1, \src, (\l1 * \strided)
+    addi            t2, \src, (\l2 * \strided)
+    addi            t3, \src, (\l3 * \strided)
+    addi            t4, \src, (\l4 * \strided)
+    vle16.v         \d1, (t1)
+    vle16.v         \d2, (t2)
+    vle16.v         \d3, (t3)
+    vle16.v         \d4, (t4)
+.endm
+
+.macro IDCT32_SEXT32 d1, d2, d3, d4, s1, s2, s3, s4
+    vsext.vf2       \d1, \s1
+    vsext.vf2       \d2, \s2
+    vsext.vf2       \d3, \s3
+    vsext.vf2       \d4, \s4
+.endm
+
+// void partialButterflyInverse32(const int16_t* src)
+.macro PARTIAL_BUTTER_FLY_INVERSE_32 src, dst, shift, strided, label
+    li              a5, 32
+    slli            a3, \strided, 2
+\label:
+    li              a4, 0
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 64, \src, 0, 8, 16, 24, v5, v7, v6, v8
+    IDCT32_LOAD_LINE 64, \src, 4, 12, 20, 28, v13, v14, v15, v16
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v1, v2, v3, v4, v5, v6, v7, v8
+
+    // v5~6:EEEE0~1 v8~9:EEEO0~1 v1~v4:EEE0~3
+    IDCT_SUMSUB_SHIFT v5, v6, v1, v2, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v3, v4, v8, v9
+    SUMSUB_ABCD     v1, v4, v2, v3, v5, v8, v6, v9
+
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 64, \src, 2, 6, 10, 14, v24, v25, v26, v27
+    IDCT32_LOAD_LINE 64, \src, 18, 22, 26, 30, v28, v29, v30, v31
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v7, v8, v9, v10, v13, v14, v15, v16
+
+    // v12~15:EEO0~3
+    IDCT_MUL_1V4IMM     89, 75, 50, 18, v7, v12, v13, v14, v15
+    IDCT_MULADD_1V4IMM  75, -18, -89, -50, v8, v12, v13, v14, v15
+    IDCT_MULADD_1V4IMM  50, -89, 18, 75, v9, v12, v13, v14, v15
+    IDCT_MULADD_1V4IMM  18, -50, 75, -89, v10, v12, v13, v14, v15
+
+    // v16~23:EE0~7
+    SUMSUB_ABCD     v16, v23, v17, v22, v1, v12, v2, v13
+    SUMSUB_ABCD     v18, v21, v19, v20, v3, v14, v4, v15
+
+    // load v9~15 7
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v9, v10, v11, v12, v24, v25, v26, v27
+    IDCT32_SEXT32   v13, v14, v15, v7, v28, v29, v30, v31
+
+    // v24~31 EO0~7
+    IDCT_MUL_1V4IMM      90, 87, 80, 70, v9, v24, v25, v26, v27
+    IDCT_MUL_1V4IMM      57, 43, 25, 9, v9, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  87, 57, 9, -43, v10, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  -80, -90, -70, -25, v10, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  80, 9, -70, -87, v11, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  -25, 57, 90, 43, v11, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  70, -43, -87, 9, v12, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  90, 25, -80, -57, v12, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  57, -80, -25, 90, v13, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  -9, -87, 43, 70, v13, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  43, -90, 57, 25, v14, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  -87, 70, 9, -80, v14, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  25, -70, 90, -80, v15, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  43, 9, -57, 87, v15, v28, v29, v30, v31
+    IDCT_MULADD_1V4IMM  9, -25, 43, -57, v7, v24, v25, v26, v27
+    IDCT_MULADD_1V4IMM  70, -80, 87, -90, v7, v28, v29, v30, v31
+
+    // v0~15 E0~15
+    SUMSUB_ABCD     v0, v15, v1, v14, v16, v24, v17, v25
+    SUMSUB_ABCD     v2, v13, v3, v12, v18, v26, v19, v27
+    SUMSUB_ABCD     v4, v11, v5, v10, v20, v28, v21, v29
+    SUMSUB_ABCD     v6, v9, v7, v8, v22, v30, v23, v31
+
+    // v17 19 21 23 O0~3
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 64, \src, 1, 3, 5, 7, v24, v25, v26, v27
+    IDCT32_LOAD_LINE 64, \src, 9, 11, 13, 15, v28, v29, v30, v31
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v16, v18, v20, v22, v24, v25, v26, v27
+    TEMP_STORE      16, v16, v18, v20, v22
+    IDCT_MUL_1V4IMM     90, 90, 88, 85, v16, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  90, 82, 67, 46, v18, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  88, 67, 31, -13, v20, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  85, 46, -13, -67, v22, v17, v19, v21, v23
+
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 64, \src, 17, 19, 21, 23, v24, v25, v26, v27
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v16, v18, v20, v22, v28, v29, v30, v31
+    TEMP_STORE      16, v16, v18, v20, v22
+    IDCT_MULADD_1V4IMM  82, 22, -54, -90, v16, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  78, -4, -82, -73, v18, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  73, -31, -90, -22, v20, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  67, -54, -78, 38, v22, v17, v19, v21, v23
+
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 64, \src, 25, 27, 29, 31, v28, v29, v30, v31
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v16, v18, v20, v22, v24, v25, v26, v27
+    TEMP_STORE      16, v16, v18, v20, v22
+    IDCT_MULADD_1V4IMM  61, -73, -46, 82, v16, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  54, -85, -4, 88, v18, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  46, -90, 38, 54, v20, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  38, -88, 73, -4, v22, v17, v19, v21, v23
+
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v16, v18, v20, v22, v28, v29, v30, v31
+    TEMP_STORE      16, v16, v18, v20, v22
+    IDCT_MULADD_1V4IMM  31, -78, 90, -61, v16, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  22, -61, 85, -90, v18, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  13, -38, 61, -78, v20, v17, v19, v21, v23
+    IDCT_MULADD_1V4IMM  4, -13, 22, -31, v22, v17, v19, v21, v23
+
+    TEMP_LOAD       16, 12, v25, v27, v29, v31
+    //v16 18 20 22 24 26 28 30 dst0~3 28~31
+    SUMSUB_ABCD v16, v30, v18, v28, v0, v17, v1, v19
+    SUMSUB_ABCD v20, v26, v22, v24, v2, v21, v3, v23
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    PBFI32_STORE8REG \dst, 0, 56, \shift, \strided
+    vsetivli        zero, 4, e32, m1, ta, ma
+
+    // v0~3 O4~7
+    TEMP_LOAD       16, 8, v16, v17, v18, v19
+    IDCT_MUL_1V4IMM     82, 78, 73, 67, v25, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  22, -4, -31, -54, v27, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -54, -82, -90, -78, v29, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -90, -73, -22, 38, v31, v0, v1, v2, v3
+
+    TEMP_LOAD       16, 4, v25, v27, v29, v31
+    IDCT_MULADD_1V4IMM  -61, 13, 78, 85, v16, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  13, 85, 67, -22, v17, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  78, 67, -38, -90, v18, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  85, -22, -90, 4, v19, v0, v1, v2, v3
+
+    TEMP_LOAD       16, 0, v16, v17, v18, v19
+    IDCT_MULADD_1V4IMM  31, -88, -13, 90, v25, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -46, -61, 82, 13, v27, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -90, 31, 61, -88, v29, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -67, 90, -46, -31, v31, v0, v1, v2, v3
+
+    IDCT_MULADD_1V4IMM  4, 54, -88, 82, v16, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  73, -38, -4, 46, v17, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  88, -90, 85, -73, v18, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  38, -46, 54, -61, v19, v0, v1, v2, v3
+
+    TEMP_LOAD       16, 12, v25, v27, v29, v31
+    //v16 18 20 22 24 26 28 30 dst4~7 24~27
+    SUMSUB_ABCD v16, v30, v18, v28, v4, v0, v5, v1
+    SUMSUB_ABCD v20, v26, v22, v24, v6, v2, v7, v3
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    PBFI32_STORE8REG \dst, 8, 48, \shift, \strided
+    vsetivli        zero, 4, e32, m1, ta, ma
+
+    // v0~7 O8~15
+    TEMP_LOAD       16, 8, v16, v17, v18, v19
+    IDCT_MUL_1V4IMM     61, 54, 46, 38, v25, v0, v1, v2, v3
+    IDCT_MUL_1V4IMM     31, 22, 13, 4, v25, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -73, -85, -90, -88, v27, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -78, -61, -38, -13, v27, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -46, -4, 38, 73, v29, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  90, 85, 61, 22, v29, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  82, 88, 54, -4, v31, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -61, -90, -78, -31, v31, v4, v5, v6, v7
+
+    TEMP_LOAD       16, 4, v25, v27, v29, v31
+    IDCT_MULADD_1V4IMM  31, -46, -90, -67, v16, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  4, 73, 88, 38, v16, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -88, -61, 31, 90, v17, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  54, -38, -90, -46, v17, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -13, 82, 61, -46, v18, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -88, -4, 85, 54, v18, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  90, 13, -88, -31, v19, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  82, 46, -73, -61, v19, v4, v5, v6, v7
+
+    TEMP_LOAD       16, 0, v16, v17, v18, v19
+    IDCT_MULADD_1V4IMM  -4, -90, 22, 85, v25, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -38, -78, 54, 67, v25, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -90, 38, 67, -78, v27, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -22, 90, -31, -73, v27, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  22, 67, -85, 13, v29, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  73, -82, 4, 78, v29, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  85, -78, 13, 61, v31, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -90, 54, 22, -82, v31, v4, v5, v6, v7
+
+    IDCT_MULADD_1V4IMM  -38, -22, 73, -90, v16, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  67, -13, -46, 85, v16, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  -78, 90, -82, 54, v17, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -13, -31, 67, -88, v17, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  54, -31, 4, 22, v18, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  -46, 67, -82, 90, v18, v4, v5, v6, v7
+    IDCT_MULADD_1V4IMM  67, -73, 78, -82, v19, v0, v1, v2, v3
+    IDCT_MULADD_1V4IMM  85, -88, 90, -90, v19, v4, v5, v6, v7
+
+    //v16 18 20 22 24 26 28 30 dst8~11 20~23
+    SUMSUB_ABCD v16, v30, v18, v28, v8, v0, v9, v1
+    SUMSUB_ABCD v20, v26, v22, v24, v10, v2, v11, v3
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    PBFI32_STORE8REG \dst, 16, 40, \shift, \strided
+    vsetivli        zero, 4, e32, m1, ta, ma
+
+    //v16 18 20 22 24 26 28 30 dst12~15 16~19
+    vsetivli         zero, 4, e32, m1, ta, ma
+    SUMSUB_ABCD v16, v30, v18, v28, v12, v4, v13, v5
+    SUMSUB_ABCD v20, v26, v22, v24, v14, v6, v15, v7
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    PBFIM_STORE8REG4 \dst, 24, \shift, \strided
+
+    add             sp, sp, a4
+    addi            a5, a5, -4
+    addi            \src, \src, 8
+    add             \dst, \dst, a3
+    bgtz            a5, \label
+    slli            t1, a3, 3
+    sub             \dst, \dst, t1
+.endm
+
+.macro PBFI32_STORE4REG dst, off1, off2, shift, strided, m1, m2, m3, m4, s1, s2, s3, s4
+    addi            t0, \dst, \off1
+    addi            t1, \dst, \off2
+    vnclip.wi       \m1, \s1, \shift
+    vnclip.wi       \m2, \s2, \shift
+    vnclip.wi       \m3, \s3, \shift
+    vnclip.wi       \m4, \s4, \shift
+    vssseg2e16.v    \m1, (t0), \strided
+    vssseg2e16.v    \m3, (t1), \strided
+.endm
+
+.macro PARTIAL_BUTTER_FLY_INVERSE_16 src, dst, shift, strided, label
+    li              a5, 16
+    slli            a3, \strided, 3
+\label:
+    li              a4, 0
+    vsetivli        zero, 8, e16, m1, ta, ma
+    IDCT32_LOAD_LINE 32, \src, 0, 4, 8, 12, v8, v10, v9, v11
+    IDCT32_LOAD_LINE 32, \src, 2, 6, 10, 14, v24, v25, v26, v27
+    vsetivli        zero, 8, e32, m2, ta, ma
+    IDCT32_SEXT32   v0, v2, v4, v6, v8, v9, v10, v11
+
+    // v8~10:EEE0~1 v12~14:EEO0~1 v0~v7:EE0~3
+    IDCT_SUMSUB_SHIFT v8, v10, v0, v2, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v4, v6, v12, v14
+    SUMSUB_ABCD     v0, v6, v2, v4, v8, v12, v10, v14
+
+    // v20~27:EO0~3
+    IDCT32_SEXT32   v12, v14, v16, v18, v24, v25, v26, v27
+    IDCT_MUL_1V4IMM     89, 75, 50, 18, v12, v20, v22, v24, v26
+    IDCT_MULADD_1V4IMM  75, -18, -89, -50, v14, v20, v22, v24, v26
+    IDCT_MULADD_1V4IMM  50, -89, 18, 75, v16, v20, v22, v24, v26
+    IDCT_MULADD_1V4IMM  18, -50, 75, -89, v18, v20, v22, v24, v26
+
+    // v8~v19 v28-v31:E0~7
+    SUMSUB_ABCD     v8, v30, v10, v28, v0, v20, v2, v22
+    SUMSUB_ABCD     v12, v18, v14, v16, v4, v24, v6, v26
+    TEMP_STORE      32, v8, v10, v12, v14
+    TEMP_STORE      32, v16, v18, v28, v30
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    IDCT32_LOAD_LINE 32, \src, 1, 3, 5, 7, v0, v1, v2, v3
+    IDCT32_LOAD_LINE 32, \src, 9, 11, 13, 15, v4, v5, v6, v7
+    vsetivli        zero, 8, e32, m2, ta, ma
+    IDCT32_SEXT32   v16, v18, v20, v22, v0, v1, v2, v3
+    IDCT32_SEXT32   v24, v26, v28, v30, v4, v5, v6, v7
+
+    // v0~15 O0~7
+    IDCT_MUL_1V4IMM      90, 87, 80, 70, v16, v0, v2, v4, v6
+    IDCT_MUL_1V4IMM      57, 43, 25, 9, v16, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  87, 57, 9, -43, v18, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  -80, -90, -70, -25, v18, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  80, 9, -70, -87, v20, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  -25, 57, 90, 43, v20, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  70, -43, -87, 9, v22, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  90, 25, -80, -57, v22, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  57, -80, -25, 90, v24, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  -9, -87, 43, 70, v24, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  43, -90, 57, 25, v26, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  -87, 70, 9, -80, v26, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  25, -70, 90, -80, v28, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  43, 9, -57, 87, v28, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  9, -25, 43, -57, v30, v0, v2, v4, v6
+    IDCT_MULADD_1V4IMM  70, -80, 87, -90, v30, v8, v10, v12, v14
+
+    // dst0~1,14~15 dst2~3,12~13
+    TEMP_LOAD       32, 4, v16, v18, v20, v22
+    SUMSUB_ABCD     v24, v30, v26, v28, v16, v0, v18, v2
+    vsetivli        zero, 8, e16, m1, ta, ma
+    PBFI32_STORE4REG \dst, 0, 28, \shift, \strided, v0, v1, v2, v3, v24, v26, v28, v30
+    vsetivli        zero, 8, e32, m2, ta, ma
+    SUMSUB_ABCD     v24, v30, v26, v28, v20, v4, v22, v6
+    vsetivli        zero, 8, e16, m1, ta, ma
+    PBFI32_STORE4REG \dst, 4, 24, \shift, \strided, v0, v1, v2, v3, v24, v26, v28, v30
+    vsetivli        zero, 8, e32, m2, ta, ma
+
+    // v16~v31 dst4~7,8~11
+    TEMP_LOAD       32, 0, v0, v2, v4, v6
+    SUMSUB_ABCD     v16, v30, v18, v28, v0, v8, v2, v10
+    SUMSUB_ABCD     v20, v26, v22, v24, v4, v12, v6, v14
+    vsetivli        zero, 8, e16, m1, ta, ma
+    PBFIM_STORE8REG8 \dst, 8, \shift, \strided
+
+    add             sp, sp, a4
+    addi            a5, a5, -8
+    addi            \src, \src, 16
+    add             \dst, \dst, a3
+    bgtz            a5, \label
+    slli            t1, a3, 1
+    sub             \dst, \dst, t1
+.endm
+
+.macro PARTIAL_BUTTER_FLY_INVERSE_8 src, dst, shift, strided, label
+    vsetivli        zero, 8, e16, m1, ta, ma
+    IDCT32_LOAD_LINE 16, \src, 0, 2, 4, 6, v8, v10, v9, v11
+    IDCT32_LOAD_LINE 16, \src, 1, 3, 5, 7, v24, v25, v26, v27
+    vsetivli        zero, 8, e32, m2, ta, ma
+    IDCT32_SEXT32   v0, v2, v4, v6, v8, v9, v10, v11
+
+    // v8~v11:EE0~1 v12~15:EO0~1 v0~7:E0~3
+    IDCT_SUMSUB_SHIFT v8, v10, v0, v2, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v4, v6, v12, v14
+    SUMSUB_ABCD v0, v6, v2, v4, v8, v12, v10, v14
+
+    // v8~15 O0~3
+    IDCT32_SEXT32   v16, v18, v20, v22, v24, v25, v26, v27
+    IDCT_MUL_1V4IMM     89, 75, 50, 18, v16, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  75, -18, -89, -50, v18, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  50, -89, 18, 75, v20, v8, v10, v12, v14
+    IDCT_MULADD_1V4IMM  18, -50, 75, -89, v22, v8, v10, v12, v14
+
+    // v16~v31  dst0~7
+    SUMSUB_ABCD     v16, v30, v18, v28, v0, v8, v2, v10
+    SUMSUB_ABCD     v20, v26, v22, v24, v4, v12, v6, v14
+    vsetivli        zero, 8, e16, m1, ta, ma
+    PBFIM_STORE8REG8 \dst, 0, \shift, \strided
+.endm
+
+.macro PARTIAL_BUTTER_FLY_INVERSE_4 src, dst, shift, strided, label
+    vsetivli        zero, 8, e16, m1, ta, ma
+    addi            t1, \src, 16
+    vle16.v         v8, (\src)
+    vle16.v         v9, (t1)
+    vslidedown.vi   v10, v8, 4
+    vslidedown.vi   v11, v9, 4
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v0, v2, v4, v6, v8, v9, v10, v11
+
+    // v8~11:E0~1 v12~15:O0~1
+    IDCT_SUMSUB_SHIFT v8, v10, v0, v2, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v4, v6, v12, v14
+
+    // v16~v23  dst0~3
+    SUMSUB_ABCD     v16, v22, v18, v20, v8, v12, v10, v14
+
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    vnclip.wi       v4, v16, \shift
+    vnclip.wi       v5, v18, \shift
+    vnclip.wi       v6, v20, \shift
+    vnclip.wi       v7, v22, \shift
+
+    vmv.v.i         v0, 5
+    TRANSPOSE_4x4   16, v8, v9, v10, v11, v4, v5, v6, v7
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    slli            t2, \strided, 1
+    add             t1, \dst, \strided
+    add             t2, t2, \dst
+    add             t3, t2, \strided
+    vse16.v         v8, (\dst)
+    vse16.v         v9, (t1)
+    vse16.v         v10, (t2)
+    vse16.v         v11, (t3)
+.endm
+
+.macro IDCT_N size
+function PFX(idct_\size\()_v)
+.if \size == 32
+    addi    sp, sp, -1024
+    addi    sp, sp, -1024
+.else
+    addi    sp, sp, -(\size * \size * 2)
+.endif
+    mv      a7, sp
+    slli    a2, a2, 1
+    li      a6, (\size * 2)
+
+    PARTIAL_BUTTER_FLY_INVERSE_\size a0, a7, 7, a6, lidct\size\()_1
+    PARTIAL_BUTTER_FLY_INVERSE_\size a7, a1, (12 - (BIT_DEPTH - 8)), a2, lidct\size\()_2
+
+.if \size == 32
+    addi    sp, sp, 1024
+    addi    sp, sp, 1024
+.else
+    addi    sp, sp, (\size * \size * 2)
+.endif
+    ret
+endfunc
+.endm
+
+// void idct16(const int16_t* src, int16_t* dst, intptr_t dstStride)
+IDCT_N 32
+IDCT_N 16
+IDCT_N 8
+IDCT_N 4
+
+.macro INVERSE_DST4 src, dst, shift, strided
+    // load v0~3
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    IDCT32_LOAD_LINE 8, \src, 0, 1, 2, 3, v5, v6, v7, v8
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT32_SEXT32   v0, v1, v2, v3, v5, v6, v7, v8
+
+    li          t0, 29
+    li          t1, 55
+    li          t2, -29
+    li          t3, 74
+
+    // v4~7 c0~3
+    vadd.vv     v4, v0, v2
+    vadd.vv     v5, v2, v3
+    vsub.vv     v6, v0, v3
+    vmul.vx     v7, v1, t3
+
+    // v8 10 12 14 dst0~3
+    vmv.v.v     v8, v7
+    vmv.v.v     v10, v7
+    vmacc.vx    v8, t0, v4
+    vmacc.vx    v8, t1, v5
+    vmacc.vx    v10, t1, v6
+    vmacc.vx    v10, t2, v5
+    vadd.vv     v12, v0, v3
+    vsub.vv     v12, v12, v2
+    vmul.vx     v12, v12, t3
+    vrsub.vi    v14, v7, 0
+    vmacc.vx    v14, t1, v4
+    vmacc.vx    v14, t0, v6
+
+    slli            t2, \strided, 1
+    add             t3, t2, \strided
+    add             t1, \dst, \strided
+    add             t2, t2, \dst
+    add             t3, t3, \dst
+
+    // store
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    vmv.v.i         v0, 5
+    vnclip.wi       v1, v8, \shift
+    vnclip.wi       v2, v10, \shift
+    vnclip.wi       v3, v12, \shift
+    vnclip.wi       v4, v14, \shift
+
+    TRANSPOSE_4x4   16, v5, v6, v7, v8, v1, v2, v3, v4
+    vsetivli        zero, 4, e16, mf2, ta, ma
+    vse16.v         v5, (\dst)
+    vse16.v         v6, (t1)
+    vse16.v         v7, (t2)
+    vse16.v         v8, (t3)
+.endm
+
+function PFX(idst4_v)
+    addi    sp, sp, -32
+    slli    a2, a2, 1
+    li      t5, 8
+
+    INVERSE_DST4 a0, sp, 7, t5
+    INVERSE_DST4 sp, a1, (12 - (BIT_DEPTH - 8)), a2
+
+    addi    sp, sp, 32
+    ret
+endfunc
+
+.macro TEMP_STORE2 r1, r2
+    addi            sp, sp, -32 * 2
+    addi            t1, sp, 32
+    addi            a4, a4, 64
+    vse32.v         \r1, (sp)
+    vse32.v         \r2, (t1)
+.endm
+
+.macro TEMP_LOAD2 num, r1, r2
+    addi            t1, sp, \num * 32
+    addi            t2, sp, (\num + 1) * 32
+    vle32.v         \r1, (t1)
+    vle32.v         \r2, (t2)
+.endm
+
+.macro DCT_STORE_2L dst, strided, line1, line2, shift, m1, m2, s1, s2
+    vnclip.wi       \m1, \s1, \shift
+    vnclip.wi       \m2, \s2, \shift
+    addi            t1, \dst, \strided * \line1
+    addi            t2, \dst, \strided * \line2
+    vse16.v         \m1, (t1)
+    vse16.v         \m2, (t2)
+.endm
+
+.macro DCT32_STORE_2L dst, strided, line1, line2, shift, m1, m2, s1, s2
+    vadd.vv         \s1, \s1, \m1
+    vadd.vv         \s2, \s2, \m2
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT_STORE_2L    \dst, \strided, \line1, \line2, \shift, \m1, \m2, \s1, \s2
+.endm
+
+.macro DCTS_STORE_4L dst, strided, l1, l2, l3, l4, shift, m1, m2, m3, m4, s1, s2, s3, s4
+    vnsrl.wi        \m1, \s1, \shift
+    vnsrl.wi        \m2, \s2, \shift
+    vnsrl.wi        \m3, \s3, \shift
+    vnsrl.wi        \m4, \s4, \shift
+    addi            t1, \dst, \strided * \l1
+    addi            t2, \dst, \strided * \l2
+    addi            t3, \dst, \strided * \l3
+    addi            t4, \dst, \strided * \l4
+    vse16.v         \m1, (t1)
+    vse16.v         \m2, (t2)
+    vse16.v         \m3, (t3)
+    vse16.v         \m4, (t4)
+.endm
+
+.macro DCT_STORE_4L dst, strided, l1, l2, l3, l4, shift, m1, m2, m3, m4, s1, s2, s3, s4
+    vnclip.wi       \m1, \s1, \shift
+    vnclip.wi       \m2, \s2, \shift
+    vnclip.wi       \m3, \s3, \shift
+    vnclip.wi       \m4, \s4, \shift
+    addi            t1, \dst, \strided * \l1
+    addi            t2, \dst, \strided * \l2
+    addi            t3, \dst, \strided * \l3
+    addi            t4, \dst, \strided * \l4
+    vse16.v         \m1, (t1)
+    vse16.v         \m2, (t2)
+    vse16.v         \m3, (t3)
+    vse16.v         \m4, (t4)
+.endm
+
+.macro DCT32_STORE_4L dst, strided, l1, l2, l3, l4, shift, m1, m2, m3, m4, s1, s2, s3, s4
+    vadd.vv         \s1, \s1, \m1
+    vadd.vv         \s2, \s2, \m2
+    vadd.vv         \s3, \s3, \m3
+    vadd.vv         \s4, \s4, \m4
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT_STORE_4L    \dst, \strided, \l1, \l2, \l3, \l4, \shift, \m1, \m2, \m3, \m4, \s1, \s2, \s3, \s4
+.endm
+
+.macro DCT32_LOAD16_LINE src, strided, i1, i2, i3, i4, d1, d2, d3, d4
+    addi            t1, \src, \i1 * 2
+    addi            t2, \src, \i2 * 2
+    addi            t3, \src, \i3 * 2
+    addi            t4, \src, \i4 * 2
+    vlse16.v        \d1, (t1), \strided
+    vlse16.v        \d2, (t2), \strided
+    vlse16.v        \d3, (t3), \strided
+    vlse16.v        \d4, (t4), \strided
+.endm
+
+.macro DCT32_SUB d1, d2, d3, d4, src1, src2, src3, src4, src5, src6, src7, src8
+    vsub.vv         \d1, \src1, \src2
+    vsub.vv         \d2, \src3, \src4
+    vsub.vv         \d3, \src5, \src6
+    vsub.vv         \d4, \src7, \src8
+.endm
+
+.macro DCT32_ADD w, d1, d2, d3, d4, src1, src2, src3, src4, src5, src6, src7, src8
+    \w\()add.vv     \d1, \src1, \src2
+    \w\()add.vv     \d2, \src3, \src4
+    \w\()add.vv     \d3, \src5, \src6
+    \w\()add.vv     \d4, \src7, \src8
+.endm
+
+.macro DCT32_MUL_1V4IMM w, m1, m2, m3, m4, src, d1, d2, d3, d4
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    \w\()mul.vx     \d1, \src, t1
+    \w\()mul.vx     \d2, \src, t2
+    \w\()mul.vx     \d3, \src, t3
+    \w\()mul.vx     \d4, \src, t4
+.endm
+
+.macro DCT32_MUL_2x1V2IMM w, m1, m2, m3, m4, s1, s2, d1, d2
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    \w\()mul.vx     \d1, \s1, t1
+    \w\()mul.vx     \d2, \s1, t2
+    \w\()macc.vx    \d1, t3, \s2
+    \w\()macc.vx    \d2, t4, \s2
+.endm
+
+.macro DCT32_MULADD_1V4IMM w, m1, m2, m3, m4, src, d1, d2, d3, d4
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    \w\()macc.vx    \d1, t1, \src
+    \w\()macc.vx    \d2, t2, \src
+    \w\()macc.vx    \d3, t3, \src
+    \w\()macc.vx    \d4, t4, \src
+.endm
+
+.macro DCT32_MULADD_2x1V2IMM w, m1, m2, m3, m4, s1, s2, d1, d2
+    li              t1, \m1
+    li              t2, \m2
+    li              t3, \m3
+    li              t4, \m4
+    \w\()macc.vx    \d1, t1, \s1
+    \w\()macc.vx    \d2, t2, \s1
+    \w\()macc.vx    \d1, t3, \s2
+    \w\()macc.vx    \d2, t4, \s2
+.endm
+
+
+.macro PARTIAL_BUTTER_FLY_32 src, dst, shift, strided, label
+    li              a5, 32
+    slli            a6, a2, 3
+\label:
+    li              a4, 0
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT32_LOAD16_LINE \src, \strided, 0, 3, 4, 7, v0, v1, v2, v3
+    DCT32_LOAD16_LINE \src, \strided, 8, 11, 12, 15, v4, v5, v6, v7
+    DCT32_LOAD16_LINE \src, \strided, 16, 19, 20, 23, v8, v9, v10, v11
+    DCT32_LOAD16_LINE \src, \strided, 24, 27, 28, 31, v12, v13, v14, v15
+
+    // O dstline 1 3 5 7
+    DCT32_SUB  v16, v17, v18, v19, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_SUB  v28, v29, v30, v31, v4, v11, v5, v10, v6, v9, v7, v8
+    DCT32_MUL_1V4IMM vw, 90, 90, 88, 85, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 85, 46, -13, -67, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 82, 22, -54, -90, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 67, -54, -78, 38, v19, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 61, -73, -46, 82, v28, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 38, -88, 73, -4, v29, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 31, -78, 90, -61, v30, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 4, -13, 22, -31, v31, v20, v22, v24, v26
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_STORE      32, v20, v22, v24, v26
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // O dstline 9 11 13 15
+    DCT32_MUL_1V4IMM vw, 82, 78, 73, 67, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -90, -73, -22, 38, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -61, 13, 78, 85, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 85, -22, -90,  4, v19, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 31, -88, -13, 90, v28, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -67, 90, -46, -31, v29, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 4, 54, -88, 82, v30, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 38, -46, 54, -61, v31, v20, v22, v24, v26
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_STORE      32, v20, v22, v24, v26
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // O dstline 17 19 21 23
+    DCT32_MUL_1V4IMM vw, 61, 54, 46, 38, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 82, 88, 54, -4, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 31, -46, -90, -67, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 90, 13, -88, -31, v19, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -4, -90, 22, 85, v28, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 85, -78, 13, 61, v29, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -38, -22, 73, -90, v30, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 67, -73, 78, -82, v31, v20, v22, v24, v26
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_STORE      32, v20, v22, v24, v26
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // O dstline 25 27 29 31
+    DCT32_MUL_1V4IMM vw, 31, 22, 13, 4, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -61, -90, -78, -31, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 4, 73, 88, 38, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 82, 46, -73, -61, v19, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -38, -78, 54, 67, v28, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -90, 54, 22, -82, v29, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 67, -13, -46, 85, v30, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 85, -88, 90, -90, v31, v20, v22, v24, v26
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_STORE      32, v20, v22, v24, v26
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // E  0 3 4 7 8 11 12 15
+    DCT32_ADD   vw, v16, v18, v20, v22, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_ADD   vw, v24, v26, v28, v30, v4, v11, v5, v10, v6, v9, v7, v8
+
+    // EE/EO 0 3 4 7
+    vsetivli        zero, 8, e32, m2, ta, ma
+    SUMSUB_ABCD v0, v8, v2, v10, v16, v30, v18, v28
+    SUMSUB_ABCD v4, v12, v6, v14, v20, v26, v22, v24
+
+    // EO dst 2 6 10 14 18 22 26 30
+    DCT32_MUL_1V4IMM v, 90, 87, 80, 70, v8, v16, v18, v20, v22
+    DCT32_MUL_1V4IMM v, 57, 43, 25, 9, v8, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 70, -43, -87, 9, v10, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 90, 25, -80, -57, v10, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 57, -80, -25, 90, v12, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, -9, -87, 43, 70, v12, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 9, -25, 43, -57, v14, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 70, -80, 87, -90, v14, v24, v26, v28, v30
+    TEMP_STORE  32, v16, v18, v20, v22
+    TEMP_STORE  32, v24, v26, v28, v30
+
+    // EEE/EEO 0 3
+    SUMSUB_ABCD v8, v12, v10, v14, v0, v6, v2, v4
+
+    // EEO dst 4 12 20 28
+    DCT32_MUL_1V4IMM v, 89, 75, 50, 18, v12, v0, v2, v4, v6
+    DCT32_MULADD_1V4IMM v, 18, -50, 75, -89, v14, v0, v2, v4, v6
+    TEMP_STORE  32, v0, v2, v4, v6
+
+    // EEEE/EEEO 0
+    vadd.vv         v16, v8, v10
+    vsub.vv         v18, v8, v10
+    TEMP_STORE2     v16, v18
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT32_LOAD16_LINE \src, \strided, 1, 2, 5, 6, v0, v1, v2, v3
+    DCT32_LOAD16_LINE \src, \strided, 9, 10, 13, 14, v4, v5, v6, v7
+    DCT32_LOAD16_LINE \src, \strided, 17, 18, 21, 22, v8, v9, v10, v11
+    DCT32_LOAD16_LINE \src, \strided, 25, 26, 29, 30, v12, v13, v14, v15
+
+    // O dstline 1 3
+    DCT32_SUB  v16, v17, v18, v19, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_SUB  v28, v29, v30, v31, v4, v11, v5, v10, v6, v9, v7, v8
+    DCT32_MUL_2x1V2IMM vw, 90, 82, 88, 67, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 78, -4, 73, -31, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 54, -85, 46, -90, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 22, -61, 13, -38, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      26, v24, v26
+    DCT32_STORE_2L  \dst, 64, 1, 3, \shift, v24, v26, v20, v22
+
+    // O dstline 5 7
+    DCT32_MUL_2x1V2IMM vw, 67, 46, 31, -13, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -82, -73, -90, -22, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -4, 88, 38, 54, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 85, -90, 61, -78, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      28, v24, v26
+    DCT32_STORE_2L  \dst, 64, 5, 7, \shift, v24, v26, v20, v22
+
+    // O dstline 9 11
+    DCT32_MUL_2x1V2IMM vw, 22, -4, -54, -82, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 13, 85, 78, 67, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -46, -61, -90, 31, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 73, -38, 88, -90, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      22, v24, v26
+    DCT32_STORE_2L  \dst, 64, 9, 11, \shift, v24, v26, v20, v22
+
+    // O dstline 13 15
+    DCT32_MUL_2x1V2IMM vw, -31, -54, -90, -78, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 67, -22, -38, -90, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 82, 13, 61, -88, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -4, 46, 85, -73, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      24, v24, v26
+    DCT32_STORE_2L  \dst, 64, 13, 15, \shift, v24, v26, v20, v22
+
+    // O dstline 17 19
+    DCT32_MUL_2x1V2IMM vw, -73, -85, -46, -4, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -88, -61, -13, 82, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -90, 38, 22, 67, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -78, 90, 54, -31, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      18, v24, v26
+    DCT32_STORE_2L  \dst, 64, 17, 19, \shift, v24, v26, v20, v22
+
+    // O dstline 21 23
+    DCT32_MUL_2x1V2IMM vw, -90, -88, 38, 73, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 31, 90, 61, -46, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 67, -78, -85, 13, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -82, 54, 4, 22, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      20, v24, v26
+    DCT32_STORE_2L  \dst, 64, 21, 23, \shift, v24, v26, v20, v22
+
+    // O dstline 25 27
+    DCT32_MUL_2x1V2IMM vw, -78, -61, 90, 85, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 54, -38, -88, -4, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -22, 90, 73, -82, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -13, -31, -46, 67, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      14, v24, v26
+    DCT32_STORE_2L  \dst, 64, 25, 27, \shift, v24, v26, v20, v22
+
+    // O dstline 29 31
+    DCT32_MUL_2x1V2IMM vw, -38, -13, 61, 22, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -90, -46, 85, 54, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -31, -73, 4, 78, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 67, -88, -82, 90, v30, v31, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD2      16, v24, v26
+    DCT32_STORE_2L  \dst, 64, 29, 31, \shift, v24, v26, v20, v22
+
+    // E  1 2 5 6 9 10 13 14
+    DCT32_ADD   vw, v16, v18, v20, v22, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_ADD   vw, v24, v26, v28, v30, v4, v11, v5, v10, v6, v9, v7, v8
+
+    // EE/EO 1 2 5 6
+    vsetivli        zero, 8, e32, m2, ta, ma
+    SUMSUB_ABCD v0, v8, v2, v10, v16, v30, v18, v28
+    SUMSUB_ABCD v4, v12, v6, v14, v20, v26, v22, v24
+
+    // EO dst 2 6 10 14 18 22 26 30
+    DCT32_MUL_1V4IMM v, 87, 57,  9, -43, v8, v16, v18, v20, v22
+    DCT32_MUL_1V4IMM v, -80, -90, -70, -25, v8, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 80,  9, -70, -87, v10, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, -25, 57, 90, 43, v10, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 43, -90, 57, 25, v12, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, -87, 70, 9, -80, v12, v24, v26, v28, v30
+    DCT32_MULADD_1V4IMM v, 25, -70, 90, -80, v14, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 43, 9, -57, 87, v14, v24, v26, v28, v30
+    TEMP_LOAD       32, 10, v8, v10, v12, v14
+    DCT32_STORE_4L  \dst, 64, 2, 6, 10, 14, \shift, v8, v10, v12, v14, v16, v18, v20, v22
+    vsetivli        zero, 8, e32, m2, ta, ma
+    TEMP_LOAD       32, 6, v8, v10, v12, v14
+    DCT32_STORE_4L  \dst, 64, 18, 22, 26, 30, \shift, v8, v10, v12, v14, v24, v26, v28, v30
+    vsetivli        zero, 8, e32, m2, ta, ma
+
+    // EEE/EEO 1 2
+    SUMSUB_ABCD v8, v12, v10, v14, v0, v6, v2, v4
+
+    // EEO dst 4 12 20 28
+    DCT32_MUL_1V4IMM v, 75, -18, -89, -50, v12, v0, v2, v4, v6
+    DCT32_MULADD_1V4IMM v, 50, -89, 18, 75, v14, v0, v2, v4, v6
+    TEMP_LOAD       32, 2, v16, v18, v20, v22
+    DCT32_STORE_4L  \dst, 64, 4, 12, 20, 28, \shift, v16, v18, v20, v22, v0, v2, v4, v6
+    vsetivli        zero, 8, e32, m2, ta, ma
+
+    // EEEE/EEEO 1
+    vadd.vv         v16, v8, v10
+    vsub.vv         v18, v8, v10
+    TEMP_LOAD2      0, v20, v22
+
+    // dst 0 16 8 24
+    IDCT_SUMSUB_SHIFT v2, v4, v20, v16, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v22, v18, v6, v8
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT_STORE_4L \dst, 64, 0, 8, 16, 24, \shift, v10, v12, v14, v16, v2, v6, v4, v8
+
+    add             sp, sp, a4
+    addi            a5, a5, -8
+    add             \src, \src, a6
+    addi            \dst, \dst, 16
+    bgtz            a5, \label
+    addi            \dst, \dst, -64
+.endm
+
+.macro PARTIAL_BUTTER_FLY_16 src, dst, shift, strided, label
+    li              a5, 16
+    slli            a6, a2, 3
+\label:
+    vsetvli         t6, a5, e16, m1, ta, ma
+    slli            t5, t6, 1
+    DCT32_LOAD16_LINE \src, \strided, 0, 1, 2, 3, v0, v1, v2, v3
+    DCT32_LOAD16_LINE \src, \strided, 4, 5, 6, 7, v4, v5, v6, v7
+    DCT32_LOAD16_LINE \src, \strided, 8, 9, 10, 11, v8, v9, v10, v11
+    DCT32_LOAD16_LINE \src, \strided, 12, 13, 14, 15, v12, v13, v14, v15
+
+    // O 0~7
+    DCT32_SUB  v16, v17, v18, v19, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_SUB  v28, v29, v30, v31, v4, v11, v5, v10, v6, v9, v7, v8
+
+    // dst 1 3
+    DCT32_MUL_2x1V2IMM vw, 90, 87, 87, 57, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 80, 9, 70, -43, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 57, -80, 43, -90, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 25, -70, 9, -25, v30, v31, v20, v22
+    DCT_STORE_2L  \dst, 32, 1, 3, \shift, v24, v26, v20, v22
+
+    // dst 5 7
+    DCT32_MUL_2x1V2IMM vw, 80, 70, 9, -43, v16, v17, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -70, -87, -87, 9, v18, v19, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, -25, 90, 57, 25, v28, v29, v20, v22
+    DCT32_MULADD_2x1V2IMM vw, 90, -80, 43, -57, v30, v31, v20, v22
+    DCT_STORE_2L  \dst, 32, 5, 7, \shift, v24, v26, v20, v22
+
+    // dst 9 11 13 15
+    DCT32_MUL_1V4IMM vw, 57, 43, 25, 9, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -80, -90, -70, -25, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -25, 57, 90, 43, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 90, 25, -80, -57, v19, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -9, -87, 43, 70, v28, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, -87, 70, 9, -80, v29, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 43, 9, -57, 87, v30, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 70, -80, 87, -90, v31, v20, v22, v24, v26
+    DCT_STORE_4L \dst, 32, 9, 11, 13, 15, \shift, v16, v18, v28, v30, v20, v22, v24, v26
+
+    // E  0~7
+    DCT32_ADD   vw, v16, v18, v20, v22, v0, v15, v1, v14, v2, v13, v3, v12
+    DCT32_ADD   vw, v24, v26, v28, v30, v4, v11, v5, v10, v6, v9, v7, v8
+
+    // EE/EO 0~3
+    vsetvli         zero, t6, e32, m2, ta, ma
+    SUMSUB_ABCD v0, v8, v2, v10, v16, v30, v18, v28
+    SUMSUB_ABCD v4, v12, v6, v14, v20, v26, v22, v24
+
+    // dst 2 6 10 14
+    DCT32_MUL_1V4IMM v, 89, 75, 50, 18, v8, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 75, -18, -89, -50, v10, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 50, -89, 18, 75, v12, v16, v18, v20, v22
+    DCT32_MULADD_1V4IMM v, 18, -50, 75, -89, v14, v16, v18, v20, v22
+    vsetvli         zero, t6, e16, m1, ta, ma
+    DCT_STORE_4L \dst, 32, 2, 6, 10, 14, \shift, v24, v26, v28, v30, v16, v18, v20, v22
+    vsetvli         zero, t6, e32, m2, ta, ma
+
+    // EEE/EEO 1 2
+    SUMSUB_ABCD v8, v12, v10, v14, v0, v6, v2, v4
+
+    // dst 0 8 4 12
+    IDCT_SUMSUB_SHIFT v16, v18, v8, v10, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v12, v14, v20, v22
+    vsetvli         zero, t6, e16, m1, ta, ma
+    DCT_STORE_4L \dst, 32, 0, 4, 8, 12, \shift, v8, v10, v12, v14, v16, v20, v18, v22
+
+    sub             a5, a5, t6
+    add             \src, \src, a6
+    add             \dst, \dst, t5
+    bgtz            a5, \label
+    addi            \dst, \dst, -32
+.endm
+
+.macro PARTIAL_BUTTER_FLY_8 src, dst, shift, strided, label
+    li              t1, \shift - 1
+    li              t2, 1
+    sll             t3, t2, t1
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vmv.v.x         v30, t3
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCT32_LOAD16_LINE \src, \strided, 0, 1, 2, 3, v0, v1, v2, v3
+    DCT32_LOAD16_LINE \src, \strided, 4, 5, 6, 7, v4, v5, v6, v7
+
+    // O 0~3
+    DCT32_SUB  v16, v17, v18, v19, v0, v7, v1, v6, v2, v5, v3, v4
+
+    // dst 1 3 5 7
+    DCT32_MUL_1V4IMM vw, 89, 75, 50, 18, v16, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 75, -18, -89, -50, v17, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 50, -89, 18, 75, v18, v20, v22, v24, v26
+    DCT32_MULADD_1V4IMM vw, 18, -50, 75, -89, v19, v20, v22, v24, v26
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vadd.vv         v20, v20, v30
+    vadd.vv         v22, v22, v30
+    vadd.vv         v24, v24, v30
+    vadd.vv         v26, v26, v30
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCTS_STORE_4L \dst, 16, 1, 3, 5, 7, \shift, v16, v18, v10, v12, v20, v22, v24, v26
+
+    // E  0~3
+    DCT32_ADD     vw, v16, v18, v20, v22, v0, v7, v1, v6, v2, v5, v3, v4
+
+    // EE/EO 0~1
+    vsetivli        zero, 8, e32, m2, ta, ma
+    SUMSUB_ABCD v0, v4, v2, v6, v16, v22, v18, v20
+
+    // dst 0 4 2 6
+    IDCT_SUMSUB_SHIFT v20, v22, v0, v2, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v4, v6, v24, v26
+    vadd.vv         v20, v20, v30
+    vadd.vv         v22, v22, v30
+    vadd.vv         v24, v24, v30
+    vadd.vv         v26, v26, v30
+    vsetivli        zero, 8, e16, m1, ta, ma
+    DCTS_STORE_4L \dst, 16, 0, 2, 4, 6, \shift, v16, v18, v10, v12, v20, v24, v22, v26
+.endm
+
+.macro PARTIAL_BUTTER_FLY_4 src, dst, shift, strided, label
+    li              t1, \shift - 1
+    li              t2, 1
+    sll             t3, t2, t1
+    vsetivli        zero, 4, e32, m1, ta, ma
+    vmv.v.x         v30, t3
+
+    vsetivli        zero, 4, e16, m1, ta, ma
+    DCT32_LOAD16_LINE \src, \strided, 0, 1, 2, 3, v0, v1, v2, v3
+
+    // E O 0~1
+    vwadd.vv        v4, v0, v3
+    vwsub.vv        v8, v0, v3
+    vwadd.vv        v6, v1, v2
+    vwsub.vv        v10, v1, v2
+
+    // dst 0~3
+    vsetivli        zero, 4, e32, m1, ta, ma
+    IDCT_SUMSUB_SHIFT v20, v22, v4, v6, 6
+    IDCT_MULADD_2x2V2IMM 83, 36, 36, -83, v8, v10, v24, v26
+    vadd.vv         v20, v20, v30
+    vadd.vv         v22, v22, v30
+    vadd.vv         v24, v24, v30
+    vadd.vv         v26, v26, v30
+    vsetivli        zero, 4, e16, m1, ta, ma
+    DCTS_STORE_4L \dst, 8, 0, 1, 2, 3, \shift, v16, v18, v10, v12, v20, v24, v22, v26
+.endm
+
+.macro DCT_N size, shiftbase
+function PFX(dct_\size\()_v)
+.if \size == 32
+    addi    sp, sp, -1024
+    addi    sp, sp, -1024
+.else
+    addi    sp, sp, -(\size * \size * 2)
+.endif
+    mv      a7, sp
+    slli    a2, a2, 1
+
+    PARTIAL_BUTTER_FLY_\size a0, a7, (\shiftbase + BIT_DEPTH - 8), a2, ldct\size\()_1
+    li      a2, (\size * 2)
+    PARTIAL_BUTTER_FLY_\size a7, a1, (\shiftbase + 7), a2, ldct\size\()_2
+
+.if \size == 32
+    addi    sp, sp, 1024
+    addi    sp, sp, 1024
+.else
+    addi    sp, sp, (\size * \size * 2)
+.endif
+    ret
+endfunc
+.endm
+
+//void dct16_neon(const int16_t *src, int16_t *dst, intptr_t srcStride)
+DCT_N 32 4
+DCT_N 16 3
+DCT_N 8 2
+DCT_N 4 1
+
+.macro FASTFORWARD_DST4 src, dst, shift, srcStride
+    // load v0~3 src0~4
+    vsetivli    zero, 4, e16, m1, ta, ma
+    addi        t1, \src, 0
+    addi        t2, \src, 2
+    addi        t3, \src, 4
+    addi        t4, \src, 6
+    addi        \src, \src, 8
+    vlse16.v    v28, (t1), \srcStride
+    vlse16.v    v29, (t2), \srcStride
+    vlse16.v    v30, (t3), \srcStride
+    vlse16.v    v31, (t4), \srcStride
+
+    vsetivli    zero, 4, e32, m1, ta, ma
+    vsext.vf2   v0, v28
+    vsext.vf2   v1, v29
+    vsext.vf2   v2, v30
+    vsext.vf2   v3, v31
+    addi        \src, \src, -8
+
+    // v4~7 c0~3
+    vadd.vv     v4, v0, v3
+    vadd.vv     v5, v1, v3
+    vsub.vv     v6, v0, v1
+    li          t1, 74
+    vmul.vx     v7, v2, t1
+
+    //v8 10 12 14 dst 0 1 2 3
+    vmv.v.v     v8, v7
+    vmv.v.v     v14, v7
+    li          t1, 29
+    li          t2, 55
+    li          t3, 74
+    li          t4, -29
+    vmacc.vx    v8, t1, v4
+    vmacc.vx    v8, t2, v5
+    vadd.vv     v10, v0, v1
+    vsub.vv     v10, v10, v3
+    vmul.vx     v10, v10, t3
+    vmul.vx     v12, v6, t1
+    vmacc.vx    v12, t2, v4
+    vsub.vv     v12, v12, v7
+    vmacc.vx    v14, t2, v6
+    vmacc.vx    v14, t4, v5
+
+    vsetivli    zero, 4, e16, m1, ta, ma
+    vnclip.wi   v20, v8, \shift
+    vnclip.wi   v21, v10, \shift
+    vnclip.wi   v22, v12, \shift
+    vnclip.wi   v23, v14, \shift
+    addi        t1, \dst, 8 * 0
+    addi        t2, \dst, 8 * 1
+    addi        t3, \dst, 8 * 2
+    addi        t4, \dst, 8 * 3
+    vse16.v     v20, (t1)
+    vse16.v     v21, (t2)
+    vse16.v     v22, (t3)
+    vse16.v     v23, (t4)
+.endm
+
+//void dst4_neon(const int16_t *src, int16_t *dst, intptr_t srcStride)
+function PFX(dst4_v)
+    addi        sp, sp, -32
+    slli        a2, a2, 1
+    mv          a5, sp
+    li          a3, 8
+
+    FASTFORWARD_DST4 a0, a5, (1 + BIT_DEPTH - 8), a2
+    FASTFORWARD_DST4 a5, a1, 8, a3
+
+    addi        sp, sp, 32
+    ret
+endfunc
+
+//void denoiseDct_c(int16_t* dctCoef, uint32_t* resSum, const uint16_t* offset, int numCoeff)
+function PFX(denoiseDct_v)
+ldenoiseDct:
+    vsetvli         t0, a3, e16, m1, ta, ma
+    slli            t5, t0, 1
+    slli            t6, t0, 2
+    vle16.v         v0, (a0)
+    vle16.v         v1, (a2)
+
+    // v2 level
+    vsetvli         zero, t0, e32, m2, ta, ma
+    vsext.vf2       v2, v0
+    // v4 sign
+    vsra.vi         v4, v2, 16
+    vsra.vi         v4, v4, 15
+    vadd.vv         v2, v2, v4
+    vxor.vv         v2, v2, v4
+    //v6 resSum
+    vle32.v         v6, (a1)
+    vadd.vv         v6, v6, v2
+    vse32.v         v6, (a1)
+
+    //v8 offset
+    vzext.vf2       v8, v1
+    vsub.vv         v2, v2, v8
+
+    //v12 dctCoef
+    vmsge.vi        v0, v2, 0
+    vxor.vv         v10, v2, v4
+    vsub.vv         v10, v10, v4
+    vmv.v.i         v12, 0
+    vsetvli         zero, t0, e32, m2, ta, mu
+    vadd.vv         v12, v12, v10, v0.t
+    vsetvli         zero, t0, e16, m1, ta, ma
+    vnsra.wi        v14, v12, 0
+    vse16.v         v14, (a0)
+
+    sub             a3, a3, t0
+    add             a0, a0, t5
+    add             a1, a1, t6
+    add             a2, a2, t5
+    bgtz            a3, ldenoiseDct
+    ret
+endfunc
+
+// void nonPsyRdoQuant_c(int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos)
+.macro NONPSYRDOQUANT_N log2TrSize
+function PFX(nonPsyRdoQuant_\log2TrSize\()_v)
+    li              t0, 1
+    slli            t0, t0, \log2TrSize + 1
+    slli            a4, a4, 1
+    add             t1, a0, a4
+    add             t2, t1, t0
+    add             t3, t2, t0
+    add             t4, t3, t0
+    vsetivli        zero, 4, e16, m1, ta, ma
+    // v1~4 signCoef
+    vle16.v         v1, (t1)
+    vle16.v         v2, (t2)
+    vle16.v         v3, (t3)
+    vle16.v         v4, (t4)
+    vwmul.vv        v6, v1, v1
+    vwmul.vv        v8, v2, v2
+    vwmul.vv        v10, v3, v3
+    vwmul.vv        v12, v4, v4
+    vsetivli        zero, 4, e64, m2, ta, ma
+    vsext.vf2       v20, v6
+    vsext.vf2       v22, v8
+    vsext.vf2       v24, v10
+    vsext.vf2       v26, v12
+.if 2 * BIT_DEPTH + 2 * \log2TrSize - 15 > 15
+    li              t5, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vmv.v.x         v2, t5
+    vsll.vv         v20, v20, v2
+    vsll.vv         v22, v22, v2
+    vsll.vv         v24, v24, v2
+    vsll.vv         v26, v26, v2
+.else
+    vsll.vi         v20, v20, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v22, v22, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v24, v24, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v26, v26, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+.endif
+    slli            a4, a4, 2
+    slli            t0, t0, 2
+    add             t1, a1, a4
+    add             t2, t1, t0
+    add             t3, t2, t0
+    add             t4, t3, t0
+    vse64.v         v20, (t1)
+    vse64.v         v22, (t2)
+    vse64.v         v24, (t3)
+    vse64.v         v26, (t4)
+    vadd.vv         v20, v20, v22
+    vadd.vv         v20, v20, v24
+    vadd.vv         v20, v20, v26
+    vmv.v.i         v2, 0
+    vredsum.vs      v20, v20, v2
+    vmv.x.s         t1, v20
+    ld              t2, (a2)
+    ld              t3, (a3)
+    add             t2, t2, t1
+    add             t3, t3, t1
+    sd              t2, (a2)
+    sd              t3, (a3)
+
+    ret
+endfunc
+.endm
+
+NONPSYRDOQUANT_N 2
+NONPSYRDOQUANT_N 3
+NONPSYRDOQUANT_N 4
+NONPSYRDOQUANT_N 5
+
+// void psyRdoQuant_c(int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos)
+.macro PSYRDOQUANT_N log2TrSize
+function PFX(PsyRdoQuant_\log2TrSize\()_v)
+    li              t0, 1
+    slli            t0, t0, \log2TrSize + 1
+    slli            a6, a6, 1
+    vsetivli        zero, 4, e16, m1, ta, ma
+    // v even(20~26) int16 signCoef
+    add             t1, a0, a6
+    add             t2, t1, t0
+    add             t3, t2, t0
+    add             t4, t3, t0
+    vle16.v         v1, (t1)
+    vle16.v         v2, (t2)
+    vle16.v         v3, (t3)
+    vle16.v         v4, (t4)
+
+    // v even(12~18) int16 predictedCoef
+    sub             t5, a1, a0
+    add             t1, t1, t5
+    add             t2, t2, t5
+    add             t3, t3, t5
+    add             t4, t4, t5
+    vle16.v         v5, (t1)
+    vle16.v         v6, (t2)
+    vle16.v         v7, (t3)
+    vle16.v         v8, (t4)
+    vwsub.vv        v12, v5, v1
+    vwsub.vv        v14, v6, v2
+    vwsub.vv        v16, v7, v3
+    vwsub.vv        v18, v8, v4
+
+    vwmul.vv        v20, v1, v1
+    vwmul.vv        v22, v2, v2
+    vwmul.vv        v24, v3, v3
+    vwmul.vv        v26, v4, v4
+
+    // v0~7 signCoef v8~15 predictedCoef
+    vsetivli        zero, 4, e64, m2, ta, ma
+    vsext.vf2       v0, v20
+    vsext.vf2       v2, v22
+    vsext.vf2       v4, v24
+    vsext.vf2       v6, v26
+    vsext.vf2       v8, v12
+    vsext.vf2       v10, v14
+    vsext.vf2       v12, v16
+    vsext.vf2       v14, v18
+
+.if 2 * BIT_DEPTH + 2 * \log2TrSize - 15 > 15
+    li              t5, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsetivli        zero, 4, e64, m2, ta, ma
+    vmv.v.x         v16, t5
+    vsll.vv         v0, v0, v16
+    vsll.vv         v2, v2, v16
+    vsll.vv         v4, v4, v16
+    vsll.vv         v6, v6, v16
+.else
+    vsll.vi         v0, v0, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v2, v2, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v4, v4, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+    vsll.vi         v6, v6, 2 * BIT_DEPTH + 2 * \log2TrSize - 15
+.endif
+
+    ld              t1, (a5)
+    vmv.v.x         v16, t1
+    vmul.vv         v8, v8, v16
+    vmul.vv         v10, v10, v16
+    vmul.vv         v12, v12, v16
+    vmul.vv         v14, v14, v16
+
+    li              t1, 2 * (15 - BIT_DEPTH - \log2TrSize) + 1
+    blez            t1, lPsyRdoQuant\log2TrSize\()1
+    vmv.v.x         v16, t1
+    vsra.vv         v8, v8, v16
+    vsra.vv         v10, v10, v16
+    vsra.vv         v12, v12, v16
+    vsra.vv         v14, v14, v16
+
+lPsyRdoQuant\log2TrSize\()1:
+    vsub.vv         v0, v0, v8
+    vsub.vv         v2, v2, v10
+    vsub.vv         v4, v4, v12
+    vsub.vv         v6, v6, v14
+
+    slli            t0, t0, 2
+    slli            a6, a6, 2
+    add             t1, a2, a6
+    add             t2, t1, t0
+    add             t3, t2, t0
+    add             t4, t3, t0
+    vse64.v         v0, (t1)
+    vse64.v         v2, (t2)
+    vse64.v         v4, (t3)
+    vse64.v         v6, (t4)
+
+    vadd.vv         v0, v0, v2
+    vadd.vv         v0, v0, v4
+    vadd.vv         v0, v0, v6
+    vmv.v.i         v2, 0
+    vredsum.vs      v0, v0, v2 
+    vmv.x.s         t1, v0
+    ld              t2, (a3)
+    ld              t3, (a4)
+    add             t2, t2, t1
+    add             t3, t3, t1
+    sd              t2, (a3)
+    sd              t3, (a4)
+
+    ret
+endfunc
+.endm
+
+PSYRDOQUANT_N 2
+PSYRDOQUANT_N 3
+PSYRDOQUANT_N 4
+PSYRDOQUANT_N 5
+

x265-4.2.tar/source/common/riscv64/filter-prim.cpp Added

@@ -0,0 +1,1280 @@
+/*****************************************************************************
+ * Copyright (C) 2021-2025 MulticoreWare, Inc
+ *
+ * Authors: Yujiao He <he.yujiao@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "filter-prim.h"
+#include <stdint.h>
+
+namespace X265_NS
+{
+#if !HIGH_BIT_DEPTH
+
+template<int coeffIdx, int width, int height>
+void interp8_horiz_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                           intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    src -= (N_TAPS / 2 - 1);
+    size_t vl = 32;
+
+    if (width > 8)
+    {
+        const vuint8m1_t shift = __riscv_vmv_v_x_u8m1(IF_FILTER_PREC, vl);
+        const vint16m2_t zero = __riscv_vmv_v_x_i16m2(0, vl);
+        vint16m2_t t0, t1, t2, t3, t4, t5, t6, t7;
+        vint16m2_t r0, r1, r2, r3, r4, r5, r6, r7;
+        vint16m2_t *s08 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7};
+        vint16m2_t *s18 = {&r0, &r1, &r2, &r3, &r4, &r5, &r6, &r7};
+
+        for (int row = 0; row < height; row +=2)
+        {
+            for (int col = 0; col < width; col += vl)
+            {
+                vl = __riscv_vsetvl_e16m2(width - col);
+
+                load_u8x16xn<8>(src + col + 0 * srcStride, s0, 1, vl);
+                load_u8x16xn<8>(src + col + 1 * srcStride, s1, 1, vl);
+
+                vint16m2_t d0 = filter8_s16x16<coeffIdx>(s0, zero, vl);
+                vint16m2_t d1 = filter8_s16x16<coeffIdx>(s1, zero, vl);
+
+                vuint16m2_t d0_u16 = __riscv_vreinterpret_v_i16m2_u16m2(__riscv_vmax_vv_i16m2(d0, zero, vl));
+                vuint16m2_t d1_u16 = __riscv_vreinterpret_v_i16m2_u16m2(__riscv_vmax_vv_i16m2(d1, zero, vl));
+
+                vuint8m1_t d0_u8 = __riscv_vnclipu_wv_u8m1(d0_u16, shift, 0, vl);
+                vuint8m1_t d1_u8 = __riscv_vnclipu_wv_u8m1(d1_u16, shift, 0, vl);
+
+                __riscv_vse8_v_u8m1(dst + col + 0 * dstStride, d0_u8, vl);
+                __riscv_vse8_v_u8m1(dst + col + 1 * dstStride, d1_u8, vl);
+            }
+
+            src += 2 * srcStride;
+            dst += 2 * dstStride;
+        }
+    }
+    else
+    {
+        vuint8mf2_t shift = __riscv_vmv_v_x_u8mf2(IF_FILTER_PREC, vl);
+        vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+        vint16m1_t t0, t1, t2, t3, t4, t5, t6, t7;
+        vint16m1_t r0, r1, r2, r3, r4, r5, r6, r7;
+        vint16m1_t *s08 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7};
+        vint16m1_t *s18 = {&r0, &r1, &r2, &r3, &r4, &r5, &r6, &r7};
+
+        for (int row = 0; row < height; row += 2)
+        {
+            int col = 0;
+            for (; col < width; col += vl)
+            {
+                vl = __riscv_vsetvl_e16m1(width - col);
+                load_u8x8xn<8>(src + col + 0 * srcStride, s0, 1, vl);
+                load_u8x8xn<8>(src + col + 1 * srcStride, s1, 1, vl);
+
+                vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, zero, vl);
+                vint16m1_t d1 = filter8_s16x8<coeffIdx>(s1, zero, vl);
+
+                vuint16m1_t d0_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d0, zero, vl));
+                vuint16m1_t d1_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d1, zero, vl));
+
+                vuint8mf2_t d0_u8 = __riscv_vnclipu_wv_u8mf2(d0_u16, shift, 0, vl);
+                vuint8mf2_t d1_u8 = __riscv_vnclipu_wv_u8mf2(d1_u16, shift, 0, vl);
+
+                __riscv_vse8_v_u8mf2(dst + col + 0 * dstStride, d0_u8, vl);
+                __riscv_vse8_v_u8mf2(dst + col + 1 * dstStride, d1_u8, vl);
+            }
+
+            src += 2 * srcStride;
+            dst += 2 * dstStride;
+        }
+    }
+}
+
+template<int coeffIdx, int width, int height>
+void interp8_vert_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                           intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    size_t vl = 16;
+
+    vuint8mf2_t shift = __riscv_vmv_v_x_u8mf2(IF_FILTER_PREC, vl);
+    vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+    vint16m1_t t0, t1, t2, t3, t4, t5, t6;
+    vint16m1_t r0, r1, r2, r3;
+    vint16m1_t *s011 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &r0, &r1, &r2, &r3};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e16m1(width - col);
+        const uint8_t *s = src;
+        uint8_t *d = dst;
+
+        load_u8x8xn<7>(s, s0, srcStride, vl);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_u8x8xn<4>(s, s0 + 7, srcStride, vl);
+            vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, zero, vl);
+            vint16m1_t d1 = filter8_s16x8<coeffIdx>(s0 + 1, zero, vl);
+            vint16m1_t d2 = filter8_s16x8<coeffIdx>(s0 + 2, zero, vl);
+            vint16m1_t d3 = filter8_s16x8<coeffIdx>(s0 + 3, zero, vl);
+
+            vuint16m1_t d0_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d0, zero, vl));
+            vuint16m1_t d1_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d1, zero, vl));
+            vuint16m1_t d2_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d2, zero, vl));
+            vuint16m1_t d3_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d3, zero, vl));
+
+            vuint8mf2_t d0_u8 = __riscv_vnclipu_wv_u8mf2(d0_u16, shift, 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnclipu_wv_u8mf2(d1_u16, shift, 0, vl);
+            vuint8mf2_t d2_u8 = __riscv_vnclipu_wv_u8mf2(d2_u16, shift, 0, vl);
+            vuint8mf2_t d3_u8 = __riscv_vnclipu_wv_u8mf2(d3_u16, shift, 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 2 * dstStride, d2_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 3 * dstStride, d3_u8, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+            *s03 = *s07;
+            *s04 = *s08;
+            *s05 = *s09;
+            *s06 = *s010;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+// Element-wise of g_chromaFilter
+const int16_t g_chromaFilter88NTAPS_CHROMA =
+{
+    { 0, 64,  0, 0 },
+    { -2, 58, 10, -2 },
+    { -4, 54, 16, -2 },
+    { -6, 46, 28, -4 },
+    { -4, 36, 36, -4 },
+    { -4, 28, 46, -6 },
+    { -2, 16, 54, -4 },
+    { -2, 10, 58, -2 }
+};
+
+template<bool coeff4, int width, int height>
+void interp4_horiz_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                           intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 4;
+    src -= (N_TAPS / 2 - 1) * 1;
+
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    size_t vl = 32;
+
+    if (width > 8)
+    {
+        vuint8m1_t shift = __riscv_vmv_v_x_u8m1(IF_FILTER_PREC, vl);
+        vint16m2_t zero = __riscv_vmv_v_x_i16m2(0, vl);
+        vint16m2_t t0, t1, t2, t3, r0, r1, r2, r3;
+        vint16m2_t *s04 = {&t0, &t1, &t2, &t3};
+        vint16m2_t *s14 = {&r0, &r1, &r2, &r3};
+
+        for (int row = 0; row < height; row +=2)
+        {
+            for (int col = 0; col < width; col += vl)
+            {
+                vl = __riscv_vsetvl_e16m2(width - col);
+
+                load_u8x16xn<4>(src + col + 0 * srcStride, s0, 1, vl);
+                load_u8x16xn<4>(src + col + 1 * srcStride, s1, 1, vl);
+                vint16m2_t d0 = filter4_s16x16<coeff4>(s0, zero, filter, vl);
+                vint16m2_t d1 = filter4_s16x16<coeff4>(s1, zero, filter, vl);
+
+                vuint16m2_t d0_u16 = __riscv_vreinterpret_v_i16m2_u16m2(__riscv_vmax_vv_i16m2(d0, zero, vl));
+                vuint16m2_t d1_u16 = __riscv_vreinterpret_v_i16m2_u16m2(__riscv_vmax_vv_i16m2(d1, zero, vl));
+
+                vuint8m1_t d0_u8 = __riscv_vnclipu_wv_u8m1(d0_u16, shift, 0, vl);
+                vuint8m1_t d1_u8 = __riscv_vnclipu_wv_u8m1(d1_u16, shift, 0, vl);
+
+                __riscv_vse8_v_u8m1(dst + col + 0 * dstStride, d0_u8, vl);
+                __riscv_vse8_v_u8m1(dst + col + 1 * dstStride, d1_u8, vl);
+            }
+
+            src += 2 * srcStride;
+            dst += 2 * dstStride;
+        }
+    }
+    else
+    {
+        vuint8mf2_t shift = __riscv_vmv_v_x_u8mf2(IF_FILTER_PREC, vl);
+        vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+        vint16m1_t t0, t1, t2, t3, r0, r1, r2, r3;
+        vint16m1_t *s04 = {&t0, &t1, &t2, &t3};
+        vint16m1_t *s14 = {&r0, &r1, &r2, &r3};
+
+        for (int row = 0; row < height; row += 2)
+        {
+            int col = 0;
+            for (; col < width; col += vl)
+            {
+                vl = __riscv_vsetvl_e16m1(width - col);
+
+                load_u8x8xn<4>(src + col + 0 * srcStride, s0, 1, vl);
+                load_u8x8xn<4>(src + col + 1 * srcStride, s1, 1, vl);
+                vint16m1_t d0 = filter4_s16x8<coeff4>(s0, zero, filter, vl);
+                vint16m1_t d1 = filter4_s16x8<coeff4>(s1, zero, filter, vl);
+
+                vuint16m1_t d0_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d0, zero, vl));
+                vuint16m1_t d1_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d1, zero, vl));
+
+                vuint8mf2_t d0_u8 = __riscv_vnclipu_wv_u8mf2(d0_u16, shift, 0, vl);
+                vuint8mf2_t d1_u8 = __riscv_vnclipu_wv_u8mf2(d1_u16, shift, 0, vl);
+
+                __riscv_vse8_v_u8mf2(dst + col + 0 * dstStride, d0_u8, vl);
+                __riscv_vse8_v_u8mf2(dst + col + 1 * dstStride, d1_u8, vl);
+            }
+
+            src += 2 * srcStride;
+            dst += 2 * dstStride;
+        }
+    }
+}
+
+template<bool coeff4, int width, int height>
+void interp4_vert_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                           intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 4;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    size_t vl = 16;
+    vuint8mf2_t shift = __riscv_vmv_v_x_u8mf2(IF_FILTER_PREC, vl);
+    vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+    vint16m1_t t0, t1, t2, t3, t4, t5, t6;
+    vint16m1_t *s07 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e16m1(width - col);
+        const uint8_t *s = src;
+        uint8_t *d = dst;
+        load_u8x8xn<3>(s, s0, srcStride, vl);
+        s += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_u8x8xn<4>(s, s0 + 3, srcStride, vl);
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, zero, filter, vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s0 + 1, zero, filter, vl);
+            vint16m1_t d2 = filter4_s16x8<coeff4>(s0 + 2, zero, filter, vl);
+            vint16m1_t d3 = filter4_s16x8<coeff4>(s0 + 3, zero, filter, vl);
+
+            vuint16m1_t d0_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d0, zero, vl));
+            vuint16m1_t d1_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d1, zero, vl));
+            vuint16m1_t d2_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d2, zero, vl));
+            vuint16m1_t d3_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d3, zero, vl));
+
+            vuint8mf2_t d0_u8 = __riscv_vnclipu_wv_u8mf2(d0_u16, shift, 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnclipu_wv_u8mf2(d1_u16, shift, 0, vl);
+            vuint8mf2_t d2_u8 = __riscv_vnclipu_wv_u8mf2(d2_u16, shift, 0, vl);
+            vuint8mf2_t d3_u8 = __riscv_vnclipu_wv_u8mf2(d3_u16, shift, 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 2 * dstStride, d2_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 3 * dstStride, d3_u8, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        if (height & 2)
+        {
+            load_u8x8xn<2>(s, s0 + 3, srcStride, vl);
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, zero, filter, vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s0 + 1, zero, filter, vl);
+
+            vuint16m1_t d0_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d0, zero, vl));
+            vuint16m1_t d1_u16 = __riscv_vreinterpret_v_i16m1_u16m1(__riscv_vmax_vv_i16m1(d1, zero, vl));
+
+            vuint8mf2_t d0_u8 = __riscv_vnclipu_wv_u8mf2(d0_u16, shift, 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnclipu_wv_u8mf2(d1_u16, shift, 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+        }
+        src += vl;
+        dst += vl;
+    }
+}
+
+
+template<int N, int width, int height>
+void interp_horiz_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_horiz_pp_rvv<1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_horiz_pp_rvv<2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_horiz_pp_rvv<3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_horiz_pp_rvv<true, width, height>(src, srcStride,
+                                                              dst, dstStride,
+                                                              coeffIdx);
+        default:
+            return interp4_horiz_pp_rvv<false, width, height>(src, srcStride,
+                                                               dst, dstStride,
+                                                               coeffIdx);
+        }
+    }
+}
+
+template<int coeffIdx, int width, int height>
+void interp8_horiz_ps_rvv(const uint8_t *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride, int isRowExt)
+{
+    int blkheight = height;
+    const int N_TAPS = 8;
+    src -= (N_TAPS / 2 - 1);
+
+    if (isRowExt)
+    {
+        src -= (N_TAPS / 2 - 1) * srcStride;
+        blkheight += N_TAPS - 1;
+    }
+
+    size_t vl = 16;
+
+    vint16m2_t c = __riscv_vmv_v_x_i16m2( -IF_INTERNAL_OFFS, vl);
+    vint16m1_t c2 = __riscv_vmv_v_x_i16m1(-IF_INTERNAL_OFFS, vl);
+
+    for (int row = 0; row + 2 <= blkheight; row += 2)
+    {
+        int col = 0;
+        for (; col + 16 <= width; col += 16)
+        {
+            vint16m2_t t0, t1, t2, t3, t4, t5, t6, t7;
+            vint16m2_t r0, r1, r2, r3, r4, r5, r6, r7;
+            vint16m2_t *s08 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7};
+            vint16m2_t *s18 = {&r0, &r1, &r2, &r3, &r4, &r5, &r6, &r7};
+
+            load_u8x16xn<8>(src + col + 0 * srcStride, s0, 1, vl);
+            load_u8x16xn<8>(src + col + 1 * srcStride, s1, 1, vl);
+            vint16m2_t d0 = filter8_s16x16<coeffIdx>(s0, c, vl);
+            vint16m2_t d1 = filter8_s16x16<coeffIdx>(s1, c, vl);
+
+            __riscv_vse16_v_i16m2(dst + col + 0 * dstStride, d0, vl);
+            __riscv_vse16_v_i16m2(dst + col + 1 * dstStride, d1, vl);
+        }
+
+        if(col < width){
+            vint16m1_t t0, t1, t2, t3, t4, t5, t6, t7;
+            vint16m1_t r0, r1, r2, r3, r4, r5, r6, r7;
+            vint16m1_t *s08 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7};
+            vint16m1_t *s18 = {&r0, &r1, &r2, &r3, &r4, &r5, &r6, &r7};
+
+            for (; col + 8 <= width; col += 8)
+            {
+                load_u8x8xn<8>(src + col + 0 * srcStride, s0, 1, 8);
+                load_u8x8xn<8>(src + col + 1 * srcStride, s1, 1, 8);
+
+                vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, c2, 8);
+                vint16m1_t d1 = filter8_s16x8<coeffIdx>(s1, c2, 8);
+
+                __riscv_vse16_v_i16m1(dst + col + 0 * dstStride, d0, 8);
+                __riscv_vse16_v_i16m1(dst + col + 1 * dstStride, d1, 8);
+            }
+
+            if (width % 8 != 0)
+            {
+                load_u8x8xn<8>(src + col + 0 * srcStride, s0, 1, 4);
+                load_u8x8xn<8>(src + col + 1 * srcStride, s1, 1, 4);
+
+                vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, c2, 4);
+                vint16m1_t d1 = filter8_s16x8<coeffIdx>(s1, c2, 4);
+
+                __riscv_vse16_v_i16m1(dst + col + 0 * dstStride, d0, 4);
+                __riscv_vse16_v_i16m1(dst + col + 1 * dstStride, d1, 4);
+            }
+        }
+
+        src += 2 * srcStride;
+        dst += 2 * dstStride;
+    }
+
+    if (isRowExt)
+    {
+        int col = 0;
+        vint16m1_t t0, t1, t2, t3, t4, t5, t6, t7;
+        vint16m1_t *s08 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &t7};
+
+        for (; col + 8 <= width; col += 8)
+        {
+            load_u8x8xn<8>(src + col, s0, 1, 8);
+            vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, c2, 8);
+            __riscv_vse16_v_i16m1(dst + col, d0, 8);
+        }
+
+        if (width % 8 != 0)
+        {
+            load_u8x8xn<8>(src + col, s0, 1, 8);
+            vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, c2, 8);
+            __riscv_vse16_v_i16m1(dst + col, d0, 4);
+        }
+    }
+}
+
+template<int coeffIdx, int width, int height>
+void interp8_vert_ps_rvv(const uint8_t *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    size_t vl = 16;
+    vint16m1_t c = __riscv_vmv_v_x_i16m1(-IF_INTERNAL_OFFS, vl);
+
+    vint16m1_t t0, t1, t2, t3, t4, t5, t6;
+    vint16m1_t r0, r1, r2, r3;
+    vint16m1_t *s011 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &r0, &r1, &r2, &r3};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e16m1(width - col);
+        const uint8_t *s = src;
+        int16_t *d = dst;
+
+        load_u8x8xn<7>(s, s0, srcStride, vl);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_u8x8xn<4>(s, s0 + 7, srcStride, vl);
+
+            vint16m1_t d0 = filter8_s16x8<coeffIdx>(s0, c, vl);
+            vint16m1_t d1 = filter8_s16x8<coeffIdx>(s0 + 1, c, vl);
+            vint16m1_t d2 = filter8_s16x8<coeffIdx>(s0 + 2, c, vl);
+            vint16m1_t d3 = filter8_s16x8<coeffIdx>(s0 + 3, c, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1, vl);
+            __riscv_vse16_v_i16m1(d + 2 * dstStride, d2, vl);
+            __riscv_vse16_v_i16m1(d + 3 * dstStride, d3, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+            *s03 = *s07;
+            *s04 = *s08;
+            *s05 = *s09;
+            *s06 = *s010;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<bool coeff4, int width, int height>
+void interp4_horiz_ps_rvv(const uint8_t *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride, int coeffIdx,
+                           int isRowExt)
+{
+    int blkheight = height;
+    const int N_TAPS = 4;
+    src -= (N_TAPS / 2 - 1);
+
+    if (isRowExt)
+    {
+        src -= (N_TAPS / 2 - 1) * srcStride;
+        blkheight += N_TAPS - 1;
+    }
+
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    size_t vl = 8;
+    const vint16m1_t c = __riscv_vmv_v_x_i16m1(-IF_INTERNAL_OFFS, vl);
+    size_t store_vl = width % 8;
+    vint16m1_t t0, t1, t2, t3, r0, r1, r2, r3;
+    vint16m1_t *s04 = {&t0, &t1, &t2, &t3};
+    vint16m1_t *s14 = {&r0, &r1, &r2, &r3};
+
+    for (int row = 0; row + 2 <= blkheight; row += 2)
+    {
+        int col = 0;
+        for (; col + 8 <= width; col += 8)
+        {
+            load_u8x8xn<4>(src + col + 0 * srcStride, s0, 1, vl);
+            load_u8x8xn<4>(src + col + 1 * srcStride, s1, 1, vl);
+
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s1, c, filter, vl);
+
+            __riscv_vse16_v_i16m1(dst + col + 0 * dstStride, d0, vl);
+            __riscv_vse16_v_i16m1(dst + col + 1 * dstStride, d1, vl);
+        }
+
+        if (width % 8 != 0)
+        {
+            load_u8x8xn<4>(src + col + 0 * srcStride, s0, 1, store_vl);
+            load_u8x8xn<4>(src + col + 1 * srcStride, s1, 1, store_vl);
+
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, store_vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s1, c, filter, store_vl);
+
+            __riscv_vse16_v_i16m1(dst + col + 0 * dstStride, d0, store_vl);
+            __riscv_vse16_v_i16m1(dst + col + 1 * dstStride, d1, store_vl);
+        }
+
+        src += 2 * srcStride;
+        dst += 2 * dstStride;
+    }
+
+    if (isRowExt)
+    {
+        int col = 0;
+        for (; col + 8 <= width; col += 8)
+        {
+            load_u8x8xn<4>(src + col, s0, 1, vl);
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, vl);
+            __riscv_vse16_v_i16m1(dst + col, d0, vl);
+        }
+
+        if (width % 8 != 0)
+        {
+            load_u8x8xn<4>(src + col, s0, 1, store_vl);
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, store_vl);
+            __riscv_vse16_v_i16m1(dst + col, d0, store_vl);
+        }
+    }
+}
+
+template<bool coeff4, int width, int height>
+void interp4_vert_ps_rvv(const uint8_t *src, intptr_t srcStride, int16_t *dst,
+                           intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 4;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    size_t vl = 16;
+    const vint16m1_t c = __riscv_vmv_v_x_i16m1(-IF_INTERNAL_OFFS, vl);
+    vint16m1_t t0, t1, t2, t3, t4, t5, t6;
+    vint16m1_t *s07 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e16m1(width - col);
+        const uint8_t *s = src;
+        int16_t *d = dst;
+
+        load_u8x8xn<3>(s, s0, srcStride, vl);
+        s += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_u8x8xn<4>(s, s0 + 3, srcStride, vl);
+
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s0 + 1, c, filter, vl);
+            vint16m1_t d2 = filter4_s16x8<coeff4>(s0 + 2, c, filter, vl);
+            vint16m1_t d3 = filter4_s16x8<coeff4>(s0 + 3, c, filter, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1, vl);
+            __riscv_vse16_v_i16m1(d + 2 * dstStride, d2, vl);
+            __riscv_vse16_v_i16m1(d + 3 * dstStride, d3, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        if(height & 2)
+        {
+            load_u8x8xn<2>(s, s0 + 3, srcStride, vl);
+
+            vint16m1_t d0 = filter4_s16x8<coeff4>(s0, c, filter, vl);
+            vint16m1_t d1 = filter4_s16x8<coeff4>(s0 + 1, c, filter, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1, vl);
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<int N, int width, int height>
+void interp_horiz_ps_rvv(const pixel *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride, int coeffIdx, int isRowExt)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_horiz_ps_rvv<1, width, height>(src, srcStride, dst,
+                                                           dstStride, isRowExt);
+        case 2:
+            return interp8_horiz_ps_rvv<2, width, height>(src, srcStride, dst,
+                                                           dstStride, isRowExt);
+        case 3:
+            return interp8_horiz_ps_rvv<3, width, height>(src, srcStride, dst,
+                                                           dstStride, isRowExt);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_horiz_ps_rvv<true, width, height>(src, srcStride,
+                                                              dst, dstStride,
+                                                              coeffIdx,
+                                                              isRowExt);
+        default:
+            return interp4_horiz_ps_rvv<false, width, height>(src, srcStride,
+                                                               dst, dstStride,
+                                                               coeffIdx,
+                                                               isRowExt);
+        }
+    }
+}
+
+template<int N, int width, int height>
+void interp_vert_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_vert_pp_rvv<1, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 2:
+            return interp8_vert_pp_rvv<2, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        case 3:
+            return interp8_vert_pp_rvv<3, width, height>(src, srcStride, dst,
+                                                           dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_vert_pp_rvv<true, width, height>(src, srcStride,
+                                                              dst, dstStride,
+                                                              coeffIdx);
+        default:
+            return interp4_vert_pp_rvv<false, width, height>(src, srcStride,
+                                                               dst, dstStride,
+                                                               coeffIdx);
+        }
+    }
+}
+
+template<int N, int width, int height>
+void interp_vert_ps_rvv(const pixel *src, intptr_t srcStride, int16_t *dst,
+                         intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_vert_ps_rvv<1, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 2:
+            return interp8_vert_ps_rvv<2, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 3:
+            return interp8_vert_ps_rvv<3, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_vert_ps_rvv<true, width, height>(src, srcStride,
+                                                             dst, dstStride,
+                                                             coeffIdx);
+        default:
+            return interp4_vert_ps_rvv<false, width, height>(src, srcStride,
+                                                              dst, dstStride,
+                                                              coeffIdx);
+        }
+    }
+}
+
+template<int coeffIdx, int width, int height>
+void interp8_vert_sp_rvv(const int16_t *src, intptr_t srcStride, pixel *dst,
+                          intptr_t dstStride)
+{
+    const int headRoom = IF_INTERNAL_PREC - X265_DEPTH;
+    const int shift0 = IF_FILTER_PREC + headRoom;
+    const int offset = (1 << (shift0 - 1)) + (IF_INTERNAL_OFFS << IF_FILTER_PREC);
+    const int N_TAPS = 8;
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+    size_t vl = 16;
+
+    vuint16m1_t shift = __riscv_vmv_v_x_u16m1(shift0, vl);
+    vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+    vint32m2_t c = __riscv_vmv_v_x_i32m2(offset, vl);
+    vint16m1_t maxVal = __riscv_vmv_v_x_i16m1((1 << X265_DEPTH) - 1, vl);
+
+    vint32m2_t t0, t1, t2, t3, t4, t5, t6;
+    vint32m2_t r0, r1, r2, r3;
+    vint32m2_t *s011 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &r0, &r1, &r2, &r3};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e32m2(width - col);
+        const int16_t *s = src;
+        uint8_t *d = dst;
+
+        load_s16x8xn<7>(s, s0, srcStride, vl);
+
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_s16x8xn<4>(s, s0 + 7, srcStride, vl);
+
+            vint32m2_t d0 = filter8_s32x8<coeffIdx>(s0, c, vl);
+            vint32m2_t d1 = filter8_s32x8<coeffIdx>(s0 + 1, c, vl);
+            vint32m2_t d2 = filter8_s32x8<coeffIdx>(s0 + 2, c, vl);
+            vint32m2_t d3 = filter8_s32x8<coeffIdx>(s0 + 3, c, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+            vint16m1_t d2_i16 = __riscv_vnclip_wv_i16m1(d2, shift, 2, vl);
+            vint16m1_t d3_i16 = __riscv_vnclip_wv_i16m1(d3, shift, 2, vl);
+
+            d0_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d0_i16, zero, vl), maxVal, vl);
+            d1_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d1_i16, zero, vl), maxVal, vl);
+            d2_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d2_i16, zero, vl), maxVal, vl);
+            d3_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d3_i16, zero, vl), maxVal, vl);
+
+            vuint8mf2_t d0_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d0_i16), 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d1_i16), 0, vl);
+            vuint8mf2_t d2_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d2_i16), 0, vl);
+            vuint8mf2_t d3_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d3_i16), 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 2 * dstStride, d2_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 3 * dstStride, d3_u8, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+            *s03 = *s07;
+            *s04 = *s08;
+            *s05 = *s09;
+            *s06 = *s010;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<bool coeff4, int width, int height>
+void interp4_vert_sp_rvv(const int16_t *src, intptr_t srcStride, uint8_t *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    assert(X265_DEPTH == 8);
+    const int headRoom = IF_INTERNAL_PREC - X265_DEPTH;
+    const int shift0 = IF_FILTER_PREC + headRoom;
+    const int offset = (1 << (shift0 - 1)) + (IF_INTERNAL_OFFS << IF_FILTER_PREC);
+    const int N_TAPS = 4;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    size_t vl = 16;
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    vuint16m1_t shift = __riscv_vmv_v_x_u16m1(shift0, vl);
+    vint16m1_t zero = __riscv_vmv_v_x_i16m1(0, vl);
+    vint32m2_t c = __riscv_vmv_v_x_i32m2(offset, vl);
+    vint16m1_t maxVal = __riscv_vmv_v_x_i16m1((1 << X265_DEPTH) - 1, vl);
+
+    vint32m2_t t0, t1, t2, t3, t4, t5, t6;
+    vint32m2_t *s07 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e32m2(width - col);
+        const int16_t *s = src;
+        uint8_t *d = dst;
+        load_s16x8xn<3>(s, s0, srcStride, vl);
+        s += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+           load_s16x8xn<4>(s, s0 + 3, srcStride, vl);
+
+            vint32m2_t d0 = filter4_s32x8<coeff4>(s0, c, filter, vl);
+            vint32m2_t d1 = filter4_s32x8<coeff4>(s0 + 1, c, filter, vl);
+            vint32m2_t d2 = filter4_s32x8<coeff4>(s0 + 2, c, filter, vl);
+            vint32m2_t d3 = filter4_s32x8<coeff4>(s0 + 3, c, filter, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+            vint16m1_t d2_i16 = __riscv_vnclip_wv_i16m1(d2, shift, 2, vl);
+            vint16m1_t d3_i16 = __riscv_vnclip_wv_i16m1(d3, shift, 2, vl);
+
+            d0_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d0_i16, zero, vl), maxVal, vl);
+            d1_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d1_i16, zero, vl), maxVal, vl);
+            d2_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d2_i16, zero, vl), maxVal, vl);
+            d3_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d3_i16, zero, vl), maxVal, vl);
+
+            vuint8mf2_t d0_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d0_i16), 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d1_i16), 0, vl);
+            vuint8mf2_t d2_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d2_i16), 0, vl);
+            vuint8mf2_t d3_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d3_i16), 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 2 * dstStride, d2_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 3 * dstStride, d3_u8, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        if(height & 2)
+        {
+            load_s16x8xn<2>(s, s0 + 3, srcStride, vl);
+
+            vint32m2_t d0 = filter4_s32x8<coeff4>(s0, c, filter, vl);
+            vint32m2_t d1 = filter4_s32x8<coeff4>(s0 + 1, c, filter, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+
+            d0_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d0_i16, zero, vl), maxVal, vl);
+            d1_i16 = __riscv_vmin_vv_i16m1(__riscv_vmax_vv_i16m1(d1_i16, zero, vl), maxVal, vl);
+
+            vuint8mf2_t d0_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d0_i16), 0, vl);
+            vuint8mf2_t d1_u8 = __riscv_vnsrl_wx_u8mf2(__riscv_vreinterpret_v_i16m1_u16m1(d1_i16), 0, vl);
+
+            __riscv_vse8_v_u8mf2(d + 0 * dstStride, d0_u8, vl);
+            __riscv_vse8_v_u8mf2(d + 1 * dstStride, d1_u8, vl);
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<int N, int width, int height>
+void interp_vert_sp_rvv(const int16_t *src, intptr_t srcStride, pixel *dst,
+                         intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_vert_sp_rvv<1, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 2:
+            return interp8_vert_sp_rvv<2, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 3:
+            return interp8_vert_sp_rvv<3, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_vert_sp_rvv<true, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx);
+        default:
+            return interp4_vert_sp_rvv<false, width, height>(src, srcStride, dst,
+                                                              dstStride, coeffIdx);
+        }
+    }
+}
+
+
+template<int coeffIdx, int width, int height>
+void interp8_vert_ss_rvv(const int16_t *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride)
+{
+    const int N_TAPS = 8;
+    src -= (N_TAPS / 2 - 1) * srcStride;
+    size_t vl = 16;
+
+    vuint16m1_t shift = __riscv_vmv_v_x_u16m1(IF_FILTER_PREC, vl);
+    vint32m2_t c = __riscv_vmv_v_x_i32m2(0, vl);
+
+    vint32m2_t t0, t1, t2, t3, t4, t5, t6;
+    vint32m2_t r0, r1, r2, r3;
+    vint32m2_t *s011 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6, &r0, &r1, &r2, &r3};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e32m2(width - col);
+        const int16_t *s = src;
+        int16_t *d = dst;
+
+        load_s16x8xn<7>(s, s0, srcStride, vl);
+        s += 7 * srcStride;
+
+        for (int row = 0; row < height; row += 4)
+        {
+            load_s16x8xn<4>(s, s0 + 7, srcStride, vl);
+
+            vint32m2_t d0 = filter8_s32x8<coeffIdx>(s0, c, vl);
+            vint32m2_t d1 = filter8_s32x8<coeffIdx>(s0 + 1, c, vl);
+            vint32m2_t d2 = filter8_s32x8<coeffIdx>(s0 + 2, c, vl);
+            vint32m2_t d3 = filter8_s32x8<coeffIdx>(s0 + 3, c, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+            vint16m1_t d2_i16 = __riscv_vnclip_wv_i16m1(d2, shift, 2, vl);
+            vint16m1_t d3_i16 = __riscv_vnclip_wv_i16m1(d3, shift, 2, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0_i16, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1_i16, vl);
+            __riscv_vse16_v_i16m1(d + 2 * dstStride, d2_i16, vl);
+            __riscv_vse16_v_i16m1(d + 3 * dstStride, d3_i16, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+            *s03 = *s07;
+            *s04 = *s08;
+            *s05 = *s09;
+            *s06 = *s010;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<bool coeff4, int width, int height>
+void interp4_vert_ss_rvv(const int16_t *src, intptr_t srcStride, int16_t *dst,
+                          intptr_t dstStride, int coeffIdx)
+{
+    const int N_TAPS = 4;
+
+    src -= (N_TAPS / 2 - 1) * srcStride;
+
+    size_t vl = 16;
+    const int16_t* filter = g_chromaFilter8coeffIdx;
+
+    vuint16m1_t shift = __riscv_vmv_v_x_u16m1(IF_FILTER_PREC, vl);
+    vint32m2_t c = __riscv_vmv_v_x_i32m2(0, vl);
+
+    vint32m2_t t0, t1, t2, t3, t4, t5, t6;
+    vint32m2_t *s07 = {&t0, &t1, &t2, &t3, &t4, &t5, &t6};
+
+    for (int col = 0; col < width; col += vl)
+    {
+        vl = __riscv_vsetvl_e32m2(width - col);
+        const int16_t *s = src;
+        int16_t *d = dst;
+        load_s16x8xn<3>(s, s0, srcStride, vl);
+        s += 3 * srcStride;
+
+        for (int row = 0; row + 4 <= height; row += 4)
+        {
+            load_s16x8xn<4>(s, s0 + 3, srcStride, vl);
+
+            vint32m2_t d0 = filter4_s32x8<coeff4>(s0, c, filter, vl);
+            vint32m2_t d1 = filter4_s32x8<coeff4>(s0 + 1, c, filter, vl);
+            vint32m2_t d2 = filter4_s32x8<coeff4>(s0 + 2, c, filter, vl);
+            vint32m2_t d3 = filter4_s32x8<coeff4>(s0 + 3, c, filter, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+            vint16m1_t d2_i16 = __riscv_vnclip_wv_i16m1(d2, shift, 2, vl);
+            vint16m1_t d3_i16 = __riscv_vnclip_wv_i16m1(d3, shift, 2, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0_i16, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1_i16, vl);
+            __riscv_vse16_v_i16m1(d + 2 * dstStride, d2_i16, vl);
+            __riscv_vse16_v_i16m1(d + 3 * dstStride, d3_i16, vl);
+
+            *s00 = *s04;
+            *s01 = *s05;
+            *s02 = *s06;
+
+            s += 4 * srcStride;
+            d += 4 * dstStride;
+        }
+
+        if(height & 2)
+        {
+            load_s16x8xn<2>(s, s0 + 3, srcStride, vl);
+
+            vint32m2_t d0 = filter4_s32x8<coeff4>(s0, c, filter, vl);
+            vint32m2_t d1 = filter4_s32x8<coeff4>(s0 + 1, c, filter, vl);
+
+            vint16m1_t d0_i16 = __riscv_vnclip_wv_i16m1(d0, shift, 2, vl);
+            vint16m1_t d1_i16 = __riscv_vnclip_wv_i16m1(d1, shift, 2, vl);
+
+            __riscv_vse16_v_i16m1(d + 0 * dstStride, d0_i16, vl);
+            __riscv_vse16_v_i16m1(d + 1 * dstStride, d1_i16, vl);
+        }
+
+        src += vl;
+        dst += vl;
+    }
+}
+
+template<int N, int width, int height>
+void interp_vert_ss_rvv(const int16_t *src, intptr_t srcStride, int16_t *dst, intptr_t dstStride, int coeffIdx)
+{
+    if (N == 8)
+    {
+        switch (coeffIdx)
+        {
+        case 1:
+            return interp8_vert_ss_rvv<1, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 2:
+            return interp8_vert_ss_rvv<2, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        case 3:
+            return interp8_vert_ss_rvv<3, width, height>(src, srcStride, dst,
+                                                          dstStride);
+        }
+    }
+    else
+    {
+        switch (coeffIdx)
+        {
+        case 4:
+            return interp4_vert_ss_rvv<true, width, height>(src, srcStride, dst,
+                                                             dstStride, coeffIdx);
+        default:
+            return interp4_vert_ss_rvv<false, width, height>(src, srcStride, dst,
+                                                              dstStride, coeffIdx);
+        }
+    }
+}
+
+template<int N, int width, int height>
+void interp_hv_pp_rvv(const pixel *src, intptr_t srcStride, pixel *dst,
+                       intptr_t dstStride, int idxX, int idxY)
+{
+    ALIGN_VAR_32(int16_t, immedwidth * (height + N - 1));
+
+    interp_horiz_ps_rvv<N, width, height>(src, srcStride, immed, width, idxX, 1);
+    interp_vert_sp_rvv<N, width, height>(immed + (N / 2 - 1) * width, width, dst,
+                                          dstStride, idxY);
+}
+
+#define CHROMA_420(W, H) \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_hpp = interp_horiz_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_hps = interp_horiz_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_vpp = interp_vert_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_vps = interp_vert_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_vsp = interp_vert_sp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I420.puCHROMA_420_ ## W ## x ## H.filter_vss = interp_vert_ss_rvv<4, W, H>;
+
+#define CHROMA_422(W, H) \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_hpp = interp_horiz_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_hps = interp_horiz_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_vpp = interp_vert_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_vps = interp_vert_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_vsp = interp_vert_sp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I422.puCHROMA_422_ ## W ## x ## H.filter_vss = interp_vert_ss_rvv<4, W, H>;
+
+#define CHROMA_444(W, H) \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_hpp = interp_horiz_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_hps = interp_horiz_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_vpp = interp_vert_pp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_vps = interp_vert_ps_rvv<4, W, H>; \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_vsp = interp_vert_sp_rvv<4, W, H>; \
+    p.chromaX265_CSP_I444.puLUMA_ ## W ## x ## H.filter_vss = interp_vert_ss_rvv<4, W, H>;
+
+#define LUMA(W, H) \
+    p.puLUMA_ ## W ## x ## H.luma_hpp     = interp_horiz_pp_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_hps     = interp_horiz_ps_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_vpp     = interp_vert_pp_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_vps     = interp_vert_ps_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_vsp     = interp_vert_sp_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_vss     = interp_vert_ss_rvv<8, W, H>; \
+    p.puLUMA_ ## W ## x ## H.luma_hvpp    = interp_hv_pp_rvv<8, W, H>;
+
+void setupFilterPrimitives_rvv(EncoderPrimitives &p)
+{
+    LUMA(4, 4);
+    LUMA(4, 8);
+    LUMA(4, 16);
+    LUMA(12, 16);
+    LUMA(8, 4);
+    LUMA(8, 8);
+    LUMA(8, 16);
+    LUMA(8, 32);
+    LUMA(16, 4);
+    LUMA(16, 8);
+    LUMA(16, 12);
+    LUMA(16, 16);
+    LUMA(16, 32);
+    LUMA(16, 64);
+    LUMA(24, 32);
+    LUMA(32, 8);
+    LUMA(32, 16);
+    LUMA(32, 24);
+    LUMA(32, 32);
+    LUMA(32, 64);
+    LUMA(48, 64);
+    LUMA(64, 16);
+    LUMA(64, 32);
+    LUMA(64, 48);
+    LUMA(64, 64);
+
+    CHROMA_420(4, 2);
+    CHROMA_420(4, 4);
+    CHROMA_420(4, 8);
+    CHROMA_420(4, 16);
+    CHROMA_420(6, 8);
+    CHROMA_420(12, 16);
+    CHROMA_420(8, 2);
+    CHROMA_420(8, 4);
+    CHROMA_420(8, 6);
+    CHROMA_420(8, 8);
+    CHROMA_420(8, 16);
+    CHROMA_420(8, 32);
+    CHROMA_420(16, 4);
+    CHROMA_420(16, 8);
+    CHROMA_420(16, 12);
+    CHROMA_420(16, 16);
+    CHROMA_420(16, 32);
+    CHROMA_420(24, 32);
+    CHROMA_420(32, 8);
+    CHROMA_420(32, 16);
+    CHROMA_420(32, 24);
+    CHROMA_420(32, 32);
+
+    CHROMA_422(4, 4);
+    CHROMA_422(4, 8);
+    CHROMA_422(4, 16);
+    CHROMA_422(4, 32);
+    CHROMA_422(6, 16);
+    CHROMA_422(12, 32);
+    CHROMA_422(8, 4);
+    CHROMA_422(8, 8);
+    CHROMA_422(8, 12);
+    CHROMA_422(8, 16);
+    CHROMA_422(8, 32);
+    CHROMA_422(8, 64);
+    CHROMA_422(16, 8);
+    CHROMA_422(16, 16);
+    CHROMA_422(16, 24);
+    CHROMA_422(16, 32);
+    CHROMA_422(16, 64);
+    CHROMA_422(24, 64);
+    CHROMA_422(32, 16);
+    CHROMA_422(32, 32);
+    CHROMA_422(32, 48);
+    CHROMA_422(32, 64);
+
+    CHROMA_444(4, 4);
+    CHROMA_444(4, 8);
+    CHROMA_444(4, 16);
+    CHROMA_444(12, 16);
+    CHROMA_444(8, 4);
+    CHROMA_444(8, 8);
+    CHROMA_444(8, 16);
+    CHROMA_444(8, 32);
+    CHROMA_444(16, 4);
+    CHROMA_444(16, 8);
+    CHROMA_444(16, 12);
+    CHROMA_444(16, 16);
+    CHROMA_444(16, 32);
+    CHROMA_444(16, 64);
+    CHROMA_444(24, 32);
+    CHROMA_444(32, 8);
+    CHROMA_444(32, 16);
+    CHROMA_444(32, 24);
+    CHROMA_444(32, 32);
+    CHROMA_444(32, 64);
+    CHROMA_444(48, 64);
+    CHROMA_444(64, 16);
+    CHROMA_444(64, 32);
+    CHROMA_444(64, 48);
+    CHROMA_444(64, 64);
+}
+#endif //!HIGH_BIT_DEPTH
+}; // namespace X265_NS

x265-4.2.tar/source/common/riscv64/filter-prim.h Added

@@ -0,0 +1,256 @@
+/*****************************************************************************
+ * Copyright (C) 2021-2025 MulticoreWare, Inc
+ *
+ * Authors: Yujiao He <he.yujiao@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include <riscv_vector.h>
+#include "primitives.h"
+
+namespace X265_NS
+{
+template<int N>
+static void inline load_u8x8xn(const uint8_t *s, vint16m1_t **d, const intptr_t stride, const size_t vl)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        *di = __riscv_vreinterpret_v_u16m1_i16m1(__riscv_vzext_vf2_u16m1(__riscv_vle8_v_u8mf2(s + i * stride, vl),vl));
+    }
+}
+
+template<int N>
+static void inline load_u8x16xn(const uint8_t *s, vint16m2_t **d, const intptr_t stride, const size_t vl)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        *di = __riscv_vreinterpret_v_u16m2_i16m2(__riscv_vzext_vf2_u16m2(__riscv_vle8_v_u8m1(s + i * stride, vl),vl));
+    }
+}
+
+template<int N>
+static void inline load_s16x8xn(const int16_t *s, vint32m2_t **d, const intptr_t stride, const size_t vl)
+{
+    for (int i = 0; i < N; ++i)
+    {
+        *di = __riscv_vsext_vf2_i32m2(__riscv_vle16_v_i16m1(s + i * stride, vl), vl);
+    }
+}
+
+#if !HIGH_BIT_DEPTH
+
+/* N_TAPS = 8 */
+template<int coeffIdx>
+vint16m1_t inline filter8_s16x8(vint16m1_t **s, const vint16m1_t c, const size_t vl)
+{
+    vint16m1_t d0;
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        d0 = __riscv_vsub_vv_i16m1(*s6, *s0, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 4, *s1, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -10, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 58, *s3, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 17, *s4, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -5, *s5, vl);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        d0 = __riscv_vmv_v_x_i16m1(0, vl);
+        vint16m1_t t0 = __riscv_vadd_vv_i16m1(*s3, *s4, vl);
+        vint16m1_t t1 = __riscv_vadd_vv_i16m1(*s2, *s5, vl);
+        vint16m1_t t2 = __riscv_vadd_vv_i16m1(*s1, *s6, vl);
+        vint16m1_t t3 = __riscv_vadd_vv_i16m1(*s0, *s7, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 40, t0, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -11, t1, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 4, t2, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -1, t3, vl);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        d0 = __riscv_vsub_vv_i16m1(*s1, *s7, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -5, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 17, *s3, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 58, *s4, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -10, *s5, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, 4, *s6, vl);
+    }
+    return __riscv_vadd_vv_i16m1(d0, c, vl);
+}
+
+template<int coeffIdx>
+vint16m2_t inline filter8_s16x16(vint16m2_t **s, const vint16m2_t c, size_t vl)
+{
+    vint16m2_t d0;
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        d0 = __riscv_vsub_vv_i16m2(*s6, *s0, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 4, *s1, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -10, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 58, *s3, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 17, *s4, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -5, *s5, vl);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        d0 = __riscv_vmv_v_x_i16m2(0, vl);
+        vint16m2_t t0 = __riscv_vadd_vv_i16m2(*s3, *s4, vl);
+        vint16m2_t t1 = __riscv_vadd_vv_i16m2(*s2, *s5, vl);
+        vint16m2_t t2 = __riscv_vadd_vv_i16m2(*s1, *s6, vl);
+        vint16m2_t t3 = __riscv_vadd_vv_i16m2(*s0, *s7, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 40, t0, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -11, t1, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 4, t2, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -1, t3, vl);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        d0 = __riscv_vsub_vv_i16m2(*s1, *s7, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -5, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 17, *s3, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 58, *s4, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -10, *s5, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, 4, *s6, vl);
+    }
+    return __riscv_vadd_vv_i16m2(d0, c, vl);
+}
+
+template<int coeffIdx>
+vint32m2_t inline filter8_s32x8(vint32m2_t **s, const vint32m2_t c, size_t vl)
+{
+    vint32m2_t d0;
+    if (coeffIdx == 1)
+    {
+        // { -1, 4, -10, 58, 17, -5, 1, 0 }
+        d0 = __riscv_vsub_vv_i32m2(*s6, *s0, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 4, *s1, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -10, *s2, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 58, *s3, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 17, *s4, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -5, *s5, vl);
+    }
+    else if (coeffIdx == 2)
+    {
+        // { -1, 4, -11, 40, 40, -11, 4, -1 }
+        d0 = __riscv_vmv_v_x_i32m2(0, vl);
+        vint32m2_t t0 = __riscv_vadd_vv_i32m2(*s3, *s4, vl);
+        vint32m2_t t1 = __riscv_vadd_vv_i32m2(*s2, *s5, vl);
+        vint32m2_t t2 = __riscv_vadd_vv_i32m2(*s1, *s6, vl);
+        vint32m2_t t3 = __riscv_vadd_vv_i32m2(*s0, *s7, vl);
+
+        d0 = __riscv_vmacc_vx_i32m2(d0, 40, t0, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -11, t1, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 4, t2, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -1, t3, vl);
+    }
+    else
+    {
+        // { 0, 1, -5, 17, 58, -10, 4, -1 }
+        d0 = __riscv_vsub_vv_i32m2(*s1, *s7, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -5, *s2, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 17, *s3, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 58, *s4, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -10, *s5, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, 4, *s6, vl);
+    }
+    return __riscv_vadd_vv_i32m2(d0, c, vl);
+}
+
+/* N_TAPS = 4 */
+template<bool coeff4>
+vint16m1_t inline filter4_s16x8(vint16m1_t **s, const vint16m1_t c, const int16_t* f, size_t vl)
+{
+    vint16m1_t d0 = __riscv_vmv_v_x_i16m1(0, vl);
+
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        vint16m1_t t1 = __riscv_vadd_vv_i16m1(*s1, *s2, vl);
+        vint16m1_t t2 = __riscv_vadd_vv_i16m1(*s0, *s3, vl);
+
+        d0 = __riscv_vmacc_vx_i16m1(d0, 36, t1, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, -4, t2, vl);
+    }
+    else
+    {
+        d0 = __riscv_vmacc_vx_i16m1(d0, f0, *s0, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, f1, *s1, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, f2, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m1(d0, f3, *s3, vl);
+    }
+    return __riscv_vadd_vv_i16m1(d0, c, vl);
+}
+
+template<bool coeff4>
+vint16m2_t inline filter4_s16x16(vint16m2_t **s, const vint16m2_t c, const int16_t* f, size_t vl)
+{
+    vint16m2_t d0 = __riscv_vmv_v_x_i16m2(0, vl);
+
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        vint16m2_t t1 = __riscv_vadd_vv_i16m2(*s1, *s2, vl);
+        vint16m2_t t2 = __riscv_vadd_vv_i16m2(*s0, *s3, vl);
+
+        d0 = __riscv_vmacc_vx_i16m2(d0, 36, t1, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, -4, t2, vl);
+    }
+    else
+    {
+        d0 = __riscv_vmacc_vx_i16m2(d0, f0, *s0, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, f1, *s1, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, f2, *s2, vl);
+        d0 = __riscv_vmacc_vx_i16m2(d0, f3, *s3, vl);
+    }
+    return __riscv_vadd_vv_i16m2(d0, c, vl);
+}
+
+template<bool coeff4>
+vint32m2_t inline filter4_s32x8(vint32m2_t **s, vint32m2_t c, const int16_t* f, size_t vl)
+{
+    vint32m2_t d0 = __riscv_vmv_v_x_i32m2(0, vl);
+
+    if (coeff4)
+    {
+        // { -4, 36, 36, -4 }
+        vint32m2_t t1 = __riscv_vadd_vv_i32m2(*s1, *s2, vl);
+        vint32m2_t t2 = __riscv_vadd_vv_i32m2(*s0, *s3, vl);
+
+        d0 = __riscv_vmacc_vx_i32m2(d0, 36, t1, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, -4, t2, vl);
+    }
+    else
+    {
+        d0 = __riscv_vmacc_vx_i32m2(d0, f0, *s0, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, f1, *s1, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, f2, *s2, vl);
+        d0 = __riscv_vmacc_vx_i32m2(d0, f3, *s3, vl);
+    }
+    return __riscv_vadd_vv_i32m2(d0, c, vl);
+    return d0;
+}
+
+#endif //!HIGH_BIT_DEPTH
+
+}; // namespace X265_NS

x265-4.2.tar/source/common/riscv64/fun-decls-prim.h Added

@@ -0,0 +1,38 @@
+/*****************************************************************************
+ * Copyright (C) 2024 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef X265_COMMON_RISCV64_INTRINSIC_SETUP_H
+#define X265_COMMON_RISCV64_INTRINSIC_SETUP_H
+
+#include "primitives.h"
+
+namespace X265_NS {
+
+void setupPixelPrimitives_rvv(EncoderPrimitives &p);
+void setupSaoPrimitives_rvv(EncoderPrimitives &p);
+void setupFilterPrimitives_rvv(EncoderPrimitives &p);
+void setupIntraPrimitives_rvv(EncoderPrimitives &p);
+
+}
+
+#endif // X265_COMMON_RISCV64_INTRINSIC_SETUP_H

x265-4.2.tar/source/common/riscv64/fun-decls.h Added

@@ -0,0 +1,203 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef _RISCV_FUNC_DECLS_
+#define _RISCV_FUNC_DECLS_
+
+#define FUNCDEF_TU(ret, name, cpu, ...) \
+    ret PFX(name ## _4x4_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _8x8_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _16x16_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _32x32_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _64x64_ ## cpu(__VA_ARGS__))
+
+#define FUNCDEF_TU_S(ret, name, cpu, ...) \
+    ret PFX(name ## _4_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _8_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _16_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _32_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _64_ ## cpu(__VA_ARGS__))
+
+#define FUNCDEF_TU_S2(ret, name, cpu, ...) \
+    ret PFX(name ## 4_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## 8_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## 16_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## 32_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## 64_ ## cpu(__VA_ARGS__))
+
+#define FUNCDEF_TU_S3(ret, name, cpu, ...) \
+    ret PFX(name ## _2_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _3_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _4_ ## cpu(__VA_ARGS__)); \
+    ret PFX(name ## _5_ ## cpu(__VA_ARGS__)); \
+
+#define FUNCDEF_PU(ret, name, cpu, ...)         \
+    ret PFX(name##_4x4_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_8x8_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_16x16_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_32x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_64x64_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_8x4_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_4x8_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_16x8_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_8x16_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_16x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_32x16_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_64x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_32x64_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_16x12_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_12x16_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_16x4_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_4x16_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_32x24_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_24x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_32x8_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_8x32_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_64x48_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_48x64_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_64x16_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_16x64_##cpu)(__VA_ARGS__)
+
+#define FUNCDEF_CHROMA_PU(ret, name, cpu, ...)  \
+    FUNCDEF_PU(ret, name, cpu, __VA_ARGS__);    \
+    ret PFX(name##_4x2_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_2x4_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_8x2_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_2x8_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_8x6_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_6x8_##cpu)(__VA_ARGS__);     \
+    ret PFX(name##_8x12_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_12x8_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_6x16_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_16x6_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_2x16_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_16x2_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_4x12_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_12x4_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_32x12_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_12x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_32x4_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_4x32_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_32x48_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_48x32_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_16x24_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_24x16_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_8x64_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_64x8_##cpu)(__VA_ARGS__);    \
+    ret PFX(name##_64x24_##cpu)(__VA_ARGS__);   \
+    ret PFX(name##_24x64_##cpu)(__VA_ARGS__);
+
+FUNCDEF_TU(int, sa8d_8x8, rvv, const pixel* pix1, intptr_t i_pix1, const pixel* pix2, intptr_t i_pix2);
+FUNCDEF_TU(int, psyCost_pp, rvv, const pixel *source, intptr_t sstride, const pixel *recon, intptr_t rstride);
+FUNCDEF_TU(int, transpose, rvv, pixel *dst, const pixel *src, intptr_t dstride, intptr_t sstride);
+
+FUNCDEF_TU_S(uint32_t, copy_cnt, v, int16_t* dst, const int16_t* src, intptr_t srcStride);
+FUNCDEF_TU_S(int, count_nonzero, v, const int16_t* quantCoeff);
+FUNCDEF_TU_S(sse_t, pixel_ssd_s, rvv, const int16_t*, intptr_t);
+FUNCDEF_TU_S(sse_t, pixel_ssd_s_aligned, rvv, const int16_t*, intptr_t);
+FUNCDEF_TU_S(void, blockfill_s, v, int16_t* dst, intptr_t dstride, int16_t val);
+FUNCDEF_TU_S(void, cpy2Dto1D_shl, v, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);
+FUNCDEF_TU_S(void, cpy2Dto1D_shr, v, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);
+FUNCDEF_TU_S(void, cpy1Dto2D_shl, v, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);
+FUNCDEF_TU_S(void, cpy1Dto2D_shr, v, int16_t* dst, const int16_t* src, intptr_t srcStride, int shift);
+FUNCDEF_TU_S(void, ssimDist, v, const pixel *fenc, uint32_t fStride, const pixel *recon, intptr_t rstride, uint64_t *ssBlock, int shift, uint64_t *ac_k);
+FUNCDEF_TU_S(void, idct, v, const int16_t* src, int16_t* dst, intptr_t dstStride);
+FUNCDEF_TU_S(void, dct, v, const int16_t* src, int16_t* dst, intptr_t srcStride);
+FUNCDEF_TU_S(void, getResidual, v, const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride);
+
+FUNCDEF_TU_S2(void, intra_pred_planar, rvv, pixel* dst, intptr_t dstride, const pixel* srcPix, int, int);
+
+FUNCDEF_TU_S3(void, nonPsyRdoQuant, v, int16_t *m_resiDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, uint32_t blkPos);
+FUNCDEF_TU_S3(void, PsyRdoQuant, v, int16_t *m_resiDctCoeff, int16_t *m_fencDctCoeff, int64_t *costUncoded, int64_t *totalUncodedCost, int64_t *totalRdCost, int64_t *psyScale, uint32_t blkPos);
+
+FUNCDEF_PU(void, pixel_avg_pp, rvv, pixel* dst, intptr_t dstride, const pixel* src0, intptr_t sstride0, const pixel* src1, intptr_t sstride1, int);
+FUNCDEF_PU(void, pixel_avg_pp_aligned, rvv, pixel* dst, intptr_t dstride, const pixel* src0, intptr_t sstride0, const pixel* src1, intptr_t sstride1, int);
+FUNCDEF_PU(void, pixel_sub_ps, v, int16_t* a, intptr_t dstride, const pixel* b0, const pixel* b1, intptr_t sstride0, intptr_t sstride1);
+FUNCDEF_PU(void, pixel_add_ps, v, pixel* a, intptr_t dstride, const pixel* b0, const int16_t* b1, intptr_t sstride0, intptr_t sstride1);
+FUNCDEF_PU(void, sad_x3, rvv, const pixel *, const pixel *, const pixel *, const pixel *, intptr_t, int32_t *);
+FUNCDEF_PU(void, sad_x4, rvv, const pixel *, const pixel *, const pixel *, const pixel *, const pixel *, intptr_t, int32_t *);
+FUNCDEF_PU(sse_t, pixel_sse_pp, rvv, const pixel*, intptr_t, const pixel*, intptr_t);
+FUNCDEF_PU(void, blockcopy_ss, v, int16_t* a, intptr_t stridea, const int16_t* b, intptr_t strideb);
+FUNCDEF_PU(void, blockcopy_sp, v, pixel* a, intptr_t stridea, const int16_t* b, intptr_t strideb);
+FUNCDEF_PU(void, blockcopy_ps, v, int16_t* a, intptr_t stridea, const pixel* b, intptr_t strideb);
+FUNCDEF_PU(uint64_t, pixel_var, v, const pixel* pix, intptr_t stride);
+FUNCDEF_PU(int, sa8d, rvv, const pixel* pix1, intptr_t i_pix1, const pixel* pix2, intptr_t i_pix2);
+
+FUNCDEF_CHROMA_PU(int, satd4, rvv, const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2);
+FUNCDEF_CHROMA_PU(int, satd8, rvv, const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2);
+FUNCDEF_CHROMA_PU(int, pixel_sad, rvv, const pixel *, intptr_t, const pixel *, intptr_t);
+FUNCDEF_CHROMA_PU(int, pixel_satd, rvv, const pixel*, intptr_t, const pixel*, intptr_t);
+FUNCDEF_CHROMA_PU(sse_t, pixel_ssd_s, rvv, const int16_t*, intptr_t);
+FUNCDEF_CHROMA_PU(sse_t, pixel_ssd_s_aligned, rvv, const int16_t*, intptr_t);
+FUNCDEF_CHROMA_PU(sse_t, pixel_sse_ss, rvv, const int16_t*, intptr_t, const int16_t*, intptr_t);
+FUNCDEF_CHROMA_PU(void, filterPixelToShort, rvv, const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride);
+FUNCDEF_CHROMA_PU(void, filterPixelToShort_aligned, rvv, const pixel* src, intptr_t srcStride, int16_t* dst, intptr_t dstStride);
+FUNCDEF_CHROMA_PU(void, addAvg, v, const int16_t* src0, const int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride);
+FUNCDEF_CHROMA_PU(void, blockcopy_pp, v, pixel* a, intptr_t stridea, const pixel* b, intptr_t strideb);
+
+void PFX(idst4_v)(const int16_t *src, int16_t *dst, intptr_t dstStride);
+void PFX(dst4_v)(const int16_t *src, int16_t *dst, intptr_t srcStride);
+void PFX(denoiseDct_v)(int16_t* dctCoef, uint32_t* resSum, const uint16_t* offset, int numCoeff);
+
+void PFX(scale1D_128to64_v(pixel *dst, const pixel *src));
+void PFX(scale2D_64to32_v(pixel* dst, const pixel* src, intptr_t stride));
+
+void PFX(dequant_scaling_v(const int16_t* quantCoef, const int32_t* deQuantCoef, int16_t* coef, int num, int per, int shift));
+void PFX(dequant_normal_v(const int16_t* quantCoef, int16_t* coef, int num, int scale, int shift));
+
+void PFX(ssim_4x4x2_core_v(const pixel* pix1, intptr_t stride1, const pixel* pix2, intptr_t stride2, int sums24));
+
+uint32_t PFX(quant_v)(const int16_t* coef, const int32_t* quantCoeff, int32_t* deltaU, int16_t* qCoef, int qBits, int add, int numCoeff);
+uint32_t PFX(nquant_v)(const int16_t* coef, const int32_t* quantCoeff, int16_t* qCoef, int qBits, int add, int numCoeff);
+
+void PFX(normFact_v)(const pixel* src, uint32_t blockSize, int shift, uint64_t *z_k);
+
+int PFX(scanPosLast_v)(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign, uint16_t *coeffFlag, uint8_t *coeffNum, int numSig, const uint16_t* scanCG4x4, const int trSize);
+uint32_t PFX(costCoeffNxN_rvv)(const uint16_t *scan, const coeff_t *coeff, intptr_t trSize, uint16_t *absCoeff, const uint8_t *tabSigCtx, uint32_t scanFlagMask, uint8_t *baseCtx, int offset, int scanPosSigOff, int subPosBase);
+
+void PFX(saoCuStatsE0_rvv)(const int16_t *diff, const pixel *rec, intptr_t stride, int endX, int endY, int32_t *stats, int32_t *count);
+void PFX(saoCuStatsE1_rvv)(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int endX, int endY, int32_t *stats, int32_t *count);
+void PFX(saoCuStatsE2_rvv)(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int8_t *upBufft, int endX, int endY, int32_t *stats, int32_t *count);
+void PFX(saoCuStatsE3_rvv)(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int endX, int endY, int32_t *stats, int32_t *count);
+
+void PFX(processSaoCUE0_rvv)(pixel * rec, int8_t * offsetEo, int width, int8_t* signLeft, intptr_t stride);
+void PFX(processSaoCUE1_rvv)(pixel* rec, int8_t* upBuff1, int8_t* offsetEo, intptr_t stride, int width);
+void PFX(processSaoCUE1_2Rows_rvv)(pixel* rec, int8_t* upBuff1, int8_t* offsetEo, intptr_t stride, int width);
+void PFX(processSaoCUE2_rvv)(pixel * rec, int8_t * bufft, int8_t * buff1, int8_t * offsetEo, int width, intptr_t stride);
+void PFX(processSaoCUE3_rvv)(pixel *rec, int8_t *upBuff1, int8_t *offsetEo, intptr_t stride, int startX, int endX);
+void PFX(processSaoCUB0_rvv)(pixel* rec, const int8_t* offset, int ctuWidth, int ctuHeight, intptr_t stride);
+void PFX(calSign_rvv)(int8_t *dst, const pixel *src1, const pixel *src2, const int endX);
+void PFX(pelFilterLumaStrong_v_rvv)(pixel* src, intptr_t srcStep, intptr_t offset, int32_t tcP, int32_t tcQ);
+void PFX(pelFilterLumaStrong_h_rvv)(pixel* src, intptr_t srcStep, intptr_t offset, int32_t tcP, int32_t tcQ);
+void PFX(pelFilterChroma_V_rvv)(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc, int32_t maskP, int32_t maskQ);
+void PFX(pelFilterChroma_H_rvv)(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc, int32_t maskP, int32_t maskQ);
+
+void PFX(weight_pp_v)(const pixel* src, pixel* dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset);
+void PFX(weight_sp_v)(const int16_t* src, pixel* dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset);
+
+void PFX(planecopy_cp_v)(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift);
+void PFX(planecopy_sp_v)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);
+void PFX(planecopy_pp_shr_v)(const pixel* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift);
+void PFX(planecopy_sp_shl_v)(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask);
+
+#endif

x265-4.2.tar/source/common/riscv64/intrapred-prim.cpp Added

@@ -0,0 +1,987 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "common.h"
+#include "primitives.h"
+
+#include "riscv64_utils.h"
+#include <riscv_vector.h>
+#include <stdint.h>
+
+namespace X265_NS
+{
+template<int tuSize>
+void intraFilter_rvv(const pixel* samples, pixel* filtered) /* 1:2:1 filtering of left and top reference samples */
+{
+    const int tuSize2 = tuSize << 1;
+    const int len = tuSize2 + tuSize2;
+
+    pixel topLeft = samples0;
+    pixel topLast = samplestuSize2;
+    pixel leftLast = sampleslen;
+
+#if !HIGH_BIT_DEPTH
+    {
+        size_t vl = __riscv_vsetvl_e16m2(len);
+        vuint16m2_t two_vec = __riscv_vmv_v_x_u16m2(2, vl);
+        for(int i = 0; i < len; i+=vl) {
+            vl = __riscv_vsetvl_e8m1(len - i);
+            vuint8m1_t sample1_u8 = __riscv_vle8_v_u8m1(&samplesi, vl);
+            vuint8m1_t sample2_u8 = __riscv_vle8_v_u8m1(&samplesi-1, vl);
+            vuint8m1_t sample3_u8 = __riscv_vle8_v_u8m1(&samplesi+1, vl);
+
+            vuint16m2_t sample1 = __riscv_vzext_vf2_u16m2(sample1_u8, vl);
+            vuint16m2_t sample2 = __riscv_vzext_vf2_u16m2(sample2_u8, vl);
+            vuint16m2_t sample3 = __riscv_vzext_vf2_u16m2(sample3_u8, vl);
+
+            vuint16m2_t result1 =  __riscv_vsll_vx_u16m2(sample1, 1, vl);
+            result1 = __riscv_vadd_vv_u16m2(result1, sample2, vl);
+            vuint16m2_t result2 =  __riscv_vadd_vv_u16m2(sample3, two_vec, vl);
+            vuint16m2_t result3 =  __riscv_vadd_vv_u16m2(result1, result2, vl);
+            result3 = __riscv_vsrl_vx_u16m2(result3, 2, vl);
+
+            vuint8m1_t result_u8 = __riscv_vnsrl_wx_u8m1(result3, 0, vl);
+            __riscv_vse8_v_u8m1(&filteredi, result_u8, vl);
+        }
+    }
+#else
+    {
+        size_t vl = __riscv_vsetvl_e16m1(len);
+        vuint16m1_t two_vec = __riscv_vmv_v_x_u16m1(2, vl);
+        for(int i = 0; i < len; i += vl) {
+            vl = __riscv_vsetvl_e16m1(len - i);
+            vuint16m1_t sample1 = __riscv_vle16_v_u16m1(&samplesi, vl);
+            vuint16m1_t sample2 = __riscv_vle16_v_u16m1(&samplesi-1, vl);
+            vuint16m1_t sample3 = __riscv_vle16_v_u16m1(&samplesi+1, vl);
+
+            vuint16m1_t result1 = __riscv_vsll_vx_u16m1(sample1, 1, vl);
+            result1 = __riscv_vadd_vv_u16m1(result1, sample2, vl);
+            vuint16m1_t result2 = __riscv_vadd_vv_u16m1(sample3, two_vec, vl);
+            vuint16m1_t result3 = __riscv_vadd_vv_u16m1(result1, result2, vl);
+
+            result3 = __riscv_vsrl_vx_u16m1(result3, 2, vl);
+            __riscv_vse16_v_u16m1(&filteredi, result3, vl);
+        }
+    }
+#endif
+    filteredtuSize2 = topLast;
+    filtered0 = ((topLeft << 1) + samples1 + samplestuSize2 + 1 + 2) >> 2;
+    filteredtuSize2 + 1 = ((samplestuSize2 + 1 << 1) + topLeft + samplestuSize2 + 2 + 2) >> 2;
+    filteredtuSize2 + tuSize2 = leftLast;
+}
+
+template<int width>
+void intra_pred_ang_rvv(pixel *dst, intptr_t dstStride, const pixel *srcPix0, int dirMode, int bFilter)
+{
+    int width2 = width << 1;
+    int horMode = dirMode < 18;
+    pixel neighbourBuf129;
+    const pixel *srcPix = srcPix0;
+
+    if (horMode) {
+        neighbourBuf0 = srcPix0;
+        if (width <= 8 && sizeof(pixel) == 1) {
+            const size_t vl = (size_t)(width << 1);
+            vuint8m1_t v1 = __riscv_vle8_v_u8m1((const uint8_t *)&srcPixwidth2 + 1, vl);
+            vuint8m1_t v2 = __riscv_vle8_v_u8m1((const uint8_t *)&srcPix1, vl);
+            __riscv_vse8_v_u8m1((uint8_t *)&neighbourBuf1, v1, vl);
+            __riscv_vse8_v_u8m1((uint8_t *)&neighbourBufwidth2 + 1, v2, vl);
+            srcPix = neighbourBuf;
+        } else {
+            memcpy(&neighbourBuf1, &srcPixwidth2 + 1, sizeof(pixel) * (width << 1));
+            memcpy(&neighbourBufwidth2 + 1, &srcPix1, sizeof(pixel) * (width << 1));
+            srcPix = neighbourBuf;
+        }
+
+    }
+
+    const int8_t angleTable17 = {-32, -26, -21, -17, -13, -9, -5, -2, 0, 2, 5, 9, 13, 17, 21, 26, 32};
+    const int16_t invAngleTable8 = {4096, 1638, 910, 630, 482, 390, 315, 256};
+
+    int angleOffset = horMode ? 10 - dirMode : dirMode - 26;
+    int angle = angleTable8 + angleOffset;
+
+    if (!angle) {
+        for (int y = 0; y < width; y++) {
+            memcpy(&dsty * dstStride, srcPix + 1, sizeof(pixel)*width);
+        }
+        if (bFilter) {
+            int topLeft = srcPix0, top = srcPix1;
+            for (int y = 0; y < width; y++) {
+                dsty * dstStride = x265_clip((int16_t)(top + ((srcPixwidth2 + 1 + y - topLeft) >> 1)));
+            }
+        }
+    } else {
+        pixel refBuf64;
+        const pixel *ref;
+
+        if (angle < 0) {
+            int nbProjected = -((width * angle) >> 5) - 1;
+            pixel *ref_pix = refBuf + nbProjected + 1;
+
+            int invAngle = invAngleTable- angleOffset - 1;
+            int invAngleSum = 128;
+
+            for (int i = 0; i < nbProjected; i++) {
+                invAngleSum += invAngle;
+                ref_pix- 2 - i = srcPixwidth2 + (invAngleSum >> 8);
+            }
+            memcpy(&ref_pix-1, srcPix, (width + 1)*sizeof(pixel));
+            ref = ref_pix;
+        } else {
+            ref = srcPix + 1;
+        }
+        int angleSum = 0;
+        for (int y = 0; y < width; y++) {
+            angleSum += angle;
+            int offset = angleSum >> 5;
+            int fraction = angleSum & 31;
+            if (fraction) {
+                size_t vl = width;
+
+                #if !HIGH_BIT_DEPTH
+                    const uint8_t *ref_u8 = (const uint8_t *)ref + offset;
+                    uint8_t *dst_u8 = (uint8_t *)dst;
+                #else
+                    const uint16_t *ref_u16 = (const uint16_t *)ref + offset;
+                    uint16_t *dst_u16 = (uint16_t *)dst;
+                #endif
+
+                switch (width) {
+                case 32: {
+                #if !HIGH_BIT_DEPTH
+                    vuint8m2_t in0 = __riscv_vle8_v_u8m2(ref_u8, vl);
+                    vuint8m2_t in1 = __riscv_vle8_v_u8m2(ref_u8 + 1, vl);
+
+                    vuint8m2_t f0 = __riscv_vmv_v_x_u8m2(32 - fraction, vl);
+                    vuint8m2_t f1 = __riscv_vmv_v_x_u8m2(fraction, vl);
+
+                    vuint16m4_t temp0 = __riscv_vwmulu_vv_u16m4(in0, f0, vl);
+                    vuint16m4_t temp1 = __riscv_vwmulu_vv_u16m4(in1, f1, vl);
+
+                    vuint16m4_t sum = __riscv_vadd_vv_u16m4(temp0, temp1, vl);
+                    vuint16m4_t sixteen = __riscv_vmv_v_x_u16m4(16, vl);
+                    sum = __riscv_vadd_vv_u16m4(sum, sixteen, vl);
+
+                    vuint8m2_t res = __riscv_vnsrl_wx_u8m2(sum, 5, vl);
+                    __riscv_vse8_v_u8m2(dst_u8 + y * dstStride, res, vl);
+                #else
+                    vuint16m2_t f0 = __riscv_vmv_v_x_u16m2(32 - fraction, vl);
+                    vuint16m2_t f1 = __riscv_vmv_v_x_u16m2(fraction, vl);
+                    vuint32m4_t sixteen = __riscv_vmv_v_x_u32m4(16, vl);
+
+                    for (int x = 0; x < width; x += vl) {
+                        vl = __riscv_vsetvl_e16m2(width - x);
+                        vuint16m2_t in0 = __riscv_vle16_v_u16m2(ref_u16 + x, vl);
+                        vuint16m2_t in1 = __riscv_vle16_v_u16m2(ref_u16 + x + 1, vl);
+
+                        vuint32m4_t temp0 = __riscv_vwmulu_vv_u32m4(in0, f0, vl);
+                        vuint32m4_t temp1 = __riscv_vwmulu_vv_u32m4(in1, f1, vl);
+                        vuint32m4_t sum = __riscv_vadd_vv_u32m4(temp0, temp1, vl);
+                        sum = __riscv_vadd_vv_u32m4(sum, sixteen, vl);
+
+                        vuint16m2_t res = __riscv_vnsrl_wx_u16m2(sum, 5, vl);
+                        __riscv_vse16_v_u16m2(dst_u16 + y * dstStride + x, res, vl);
+                    }
+                #endif
+
+                }
+                break;
+                case 16: {
+                #if !HIGH_BIT_DEPTH
+                    vuint8m1_t in0 = __riscv_vle8_v_u8m1(ref_u8, vl);
+                    vuint8m1_t in1 = __riscv_vle8_v_u8m1(ref_u8 + 1, vl);
+
+                    vuint8m1_t f0 = __riscv_vmv_v_x_u8m1(32 - fraction, vl);
+                    vuint8m1_t f1 = __riscv_vmv_v_x_u8m1(fraction, vl);
+
+                    vuint16m2_t temp0 = __riscv_vwmulu_vv_u16m2(in0, f0, vl);
+                    vuint16m2_t temp1 = __riscv_vwmulu_vv_u16m2(in1, f1, vl);
+                    vuint16m2_t sum = __riscv_vadd_vv_u16m2(temp0, temp1, vl);
+                    vuint16m2_t sixteen = __riscv_vmv_v_x_u16m2(16, vl);
+                    sum = __riscv_vadd_vv_u16m2(sum, sixteen, vl);
+
+                    vuint8m1_t res = __riscv_vnsrl_wx_u8m1(sum, 5, vl);
+                    __riscv_vse8_v_u8m1(dst_u8 + y * dstStride, res, vl);
+                #else
+                    vuint16m2_t in0 = __riscv_vle16_v_u16m2(ref_u16, vl);
+                    vuint16m2_t in1 = __riscv_vle16_v_u16m2(ref_u16 + 1, vl);
+
+                    vuint16m2_t f0 = __riscv_vmv_v_x_u16m2(32 - fraction, vl);
+                    vuint16m2_t f1 = __riscv_vmv_v_x_u16m2(fraction, vl);
+
+                    vuint32m4_t temp0 = __riscv_vwmulu_vv_u32m4(in0, f0, vl);
+                    vuint32m4_t temp1 = __riscv_vwmulu_vv_u32m4(in1, f1, vl);
+
+                    vuint32m4_t sum = __riscv_vadd_vv_u32m4(temp0, temp1, vl);
+                    vuint16m2_t sixteen = __riscv_vmv_v_x_u16m2(16, vl);
+                    sum = __riscv_vwaddu_wv_u32m4(sum, sixteen, vl);
+
+                    vuint16m2_t res = __riscv_vnsrl_wx_u16m2(sum, 5, vl);
+                    __riscv_vse16_v_u16m2(dst_u16 + y * dstStride, res, vl);
+                #endif
+                }
+                break;
+                case 8: {
+                #if !HIGH_BIT_DEPTH
+                    vuint8mf2_t in0 = __riscv_vle8_v_u8mf2(ref_u8, vl);
+                    vuint8mf2_t in1 = __riscv_vle8_v_u8mf2(ref_u8 + 1, vl);
+
+                    vuint8mf2_t f0 = __riscv_vmv_v_x_u8mf2(32 - fraction, vl);
+                    vuint8mf2_t f1 = __riscv_vmv_v_x_u8mf2(fraction, vl);
+
+                    vuint16m1_t temp0 = __riscv_vwmulu_vv_u16m1(in0, f0, vl);
+                    vuint16m1_t temp1 = __riscv_vwmulu_vv_u16m1(in1, f1, vl);
+
+                    vuint16m1_t sum = __riscv_vadd_vv_u16m1(temp0, temp1, vl);
+                    vuint16m1_t sixteen = __riscv_vmv_v_x_u16m1(16, vl);
+                    sum = __riscv_vadd_vv_u16m1(sum, sixteen, vl);
+
+                    vuint8mf2_t res = __riscv_vnsrl_wx_u8mf2(sum, 5, vl);
+                    __riscv_vse8_v_u8mf2(dst_u8 + y * dstStride, res, vl);
+                #else
+                    vuint16m1_t in0 = __riscv_vle16_v_u16m1(ref_u16, vl);
+                    vuint16m1_t in1 = __riscv_vle16_v_u16m1(ref_u16 + 1, vl);
+
+                    vuint16m1_t f0 = __riscv_vmv_v_x_u16m1(32 - fraction, vl);
+                    vuint16m1_t f1 = __riscv_vmv_v_x_u16m1(fraction, vl);
+
+                    vuint32m2_t temp0 = __riscv_vwmulu_vv_u32m2(in0, f0, vl);
+                    vuint32m2_t temp1 = __riscv_vwmulu_vv_u32m2(in1, f1, vl);
+
+                    vuint32m2_t sum = __riscv_vadd_vv_u32m2(temp0, temp1, vl);
+                    vuint16m1_t sixteen = __riscv_vmv_v_x_u16m1(16, vl);
+                    sum = __riscv_vwaddu_wv_u32m2(sum, sixteen, vl);
+
+                    vuint16m1_t res = __riscv_vnsrl_wx_u16m1(sum, 5, vl);
+                    __riscv_vse16_v_u16m1(dst_u16 + y * dstStride, res, vl);
+                #endif
+
+                }
+                break;
+                case 4: {
+                #if !HIGH_BIT_DEPTH
+                    vuint8mf4_t in0 = __riscv_vle8_v_u8mf4(ref_u8, vl);
+                    vuint8mf4_t in1 = __riscv_vle8_v_u8mf4(ref_u8 + 1, vl);
+                    vuint16mf2_t sixteen = __riscv_vmv_v_x_u16mf2(16, vl);
+
+                    vuint8mf4_t f0 = __riscv_vmv_v_x_u8mf4(32 - fraction, vl);
+                    vuint8mf4_t f1 = __riscv_vmv_v_x_u8mf4(fraction, vl);
+
+                    vuint16mf2_t temp0 = __riscv_vwmulu_vv_u16mf2(in0, f0, vl);
+                    vuint16mf2_t temp1 = __riscv_vwmulu_vv_u16mf2(in1, f1, vl);
+                    vuint16mf2_t temp = __riscv_vadd_vv_u16mf2(temp0, temp1, vl);
+
+                    vuint16mf2_t sum = __riscv_vadd_vv_u16mf2(temp, sixteen, vl);
+                    vuint8mf4_t res = __riscv_vnsrl_wx_u8mf4(sum, 5, vl);
+                    __riscv_vse8_v_u8mf4(dst_u8 + y * dstStride, res, vl);
+                #else
+                    vuint16mf2_t in0 = __riscv_vle16_v_u16mf2(ref_u16, vl);
+                    vuint16mf2_t in1 = __riscv_vle16_v_u16mf2(ref_u16 + 1, vl);
+
+                    vuint16mf2_t f0 = __riscv_vmv_v_x_u16mf2(32 - fraction, vl);
+                    vuint16mf2_t f1 = __riscv_vmv_v_x_u16mf2(fraction, vl);
+
+                    vuint32m1_t temp0 = __riscv_vwmulu_vv_u32m1(in0, f0, vl);
+                    vuint32m1_t temp1 = __riscv_vwmulu_vv_u32m1(in1, f1, vl);
+
+                    vuint32m1_t sum = __riscv_vadd_vv_u32m1(temp0, temp1, vl);
+                    vuint16mf2_t sixteen = __riscv_vmv_v_x_u16mf2(16, vl);
+                    sum = __riscv_vwaddu_wv_u32m1(sum, sixteen, vl);
+
+                    vuint16mf2_t res = __riscv_vnsrl_wx_u16mf2(sum, 5, vl);
+                    __riscv_vse16_v_u16mf2(dst_u16 + y * dstStride, res, vl);
+                #endif
+                }
+                break;
+                default: {
+                    for (int x = 0; x < width; x++) {
+                        dsty * dstStride + x = (pixel)(((32 - fraction) * refoffset + x + fraction * refoffset + x + 1 + 16) >> 5);
+                    }
+                }
+                break;
+                }
+            } else {
+                memcpy(&dsty * dstStride, &refoffset, sizeof(pixel)*width);
+            }
+        }
+    }
+    if (horMode) {
+        if (width == 4) {
+            transpose4x4_rvv(dst, dst, dstStride, dstStride);
+        } else if (width == 8) {
+            transpose8x8_rvv(dst, dst, dstStride, dstStride);
+        } else if (width == 16) {
+            transpose16x16_rvv(dst, dst, dstStride, dstStride);
+        } else if (width == 32) {
+            transpose32x32_rvv(dst, dst, dstStride, dstStride);
+        } else {
+            for (int y = 0; y < width - 1; y++) {
+                for (int x = y + 1; x < width; x++) {
+                    pixel tmp = dsty * dstStride + x;
+                    dsty * dstStride + x = dstx * dstStride + y;
+                    dstx * dstStride + y = tmp;
+                }
+            }
+        }
+    }
+
+}
+
+template<int log2Size>
+void all_angs_pred_rvv(pixel *dest, pixel *refPix, pixel *filtPix, int bLuma) {
+    const int size = 1 << log2Size;
+    for (int mode = 2; mode <= 34; mode++) {
+        pixel *srcPix  = (g_intraFilterFlagsmode & size ? filtPix  : refPix);
+        pixel *out = dest + ((mode - 2) << (log2Size * 2));
+        intra_pred_ang_rvv<size>(out, size, srcPix, mode, bLuma);
+
+        bool modeHor = (mode < 18);
+        if (modeHor) {
+            if (size == 4) {
+                transpose4x4_rvv(out, out, size, size);
+            } else if (size == 8) {
+                transpose8x8_rvv(out, out, size, size);
+            } else if (size == 16) {
+                transpose16x16_rvv(out, out, size, size);
+            } else if (size == 32) {
+                transpose32x32_rvv(out, out, size, size);
+            } else {
+                for (int k = 0; k < size - 1; k++) {
+                    for (int l = k + 1; l < size; l++) {
+                        pixel tmp = outk * size + l;
+                        outk * size + l = outl * size + k;
+                        outl * size + k = tmp;
+                    }
+                }
+            }
+        }
+    }
+}
+
+#if HIGH_BIT_DEPTH
+template<int log2Size>
+void planar_pred_rvv(pixel * dst, intptr_t dstStride, const pixel * srcPix, int /*dirMode*/, int /*bFilter*/)
+{
+    const int blkSize = 1 << log2Size;
+
+    const pixel* above = srcPix + 1;
+    const pixel* left = srcPix + (2 * blkSize + 1);
+
+    switch (blkSize) {
+    case 4:
+    {
+        size_t vl = blkSize;
+        vuint16mf2_t above0 = __riscv_vle16_v_u16mf2(above, vl);
+
+        vuint16mf2_t topRight = __riscv_vmv_v_x_u16mf2(aboveblkSize, vl);
+        vuint16mf2_t bottomLeft = __riscv_vmv_v_x_u16mf2(leftblkSize, vl);
+
+        vuint16mf2_t cx = __riscv_vmv_v_x_u16mf2(blkSize - 1, vl);
+        vuint16mf2_t oneVec = __riscv_vmv_v_x_u16mf2(1, vl);
+        vuint16mf2_t blkSizeVec = __riscv_vmv_v_x_u16mf2(blkSize, vl);
+
+        // topRight constant
+        vuint16mf2_t inc = __riscv_vid_v_u16mf2(vl);
+        vuint16mf2_t tR0 = __riscv_vadd_vv_u16mf2(inc, oneVec, vl);
+
+        // left constant
+        vuint16mf2_t l0 = __riscv_vsub_vv_u16mf2(blkSizeVec, tR0, vl);
+
+        vuint16mf2_t offset_bottomLeft = __riscv_vadd_vv_u16mf2(blkSizeVec, bottomLeft, vl);
+
+        vuint16mf2_t t0 = __riscv_vmacc_vv_u16mf2(offset_bottomLeft, topRight, tR0, vl);
+        t0 = __riscv_vmacc_vv_u16mf2(t0, above0, cx, vl);
+
+        vuint16mf2_t sub_bottomLeft_above0 = __riscv_vsub_vv_u16mf2(bottomLeft, above0, vl);
+
+        for (int y = 0; y < blkSize; y++) {
+            vuint16mf2_t left_v = __riscv_vmv_v_x_u16mf2(lefty, vl);
+            vuint16mf2_t r0 = __riscv_vmacc_vv_u16mf2(t0, left_v, l0, vl);
+            vuint16mf2_t d0 = __riscv_vsrl_vx_u16mf2(r0, log2Size + 1, vl);
+            __riscv_vse16_v_u16mf2(dst + y * dstStride, d0, vl);
+            t0 = __riscv_vadd_vv_u16mf2(t0, sub_bottomLeft_above0, vl);
+        }
+    }
+    break;
+    case 8:
+    {
+        size_t vl = blkSize;
+        vuint16m1_t above0 = __riscv_vle16_v_u16m1(above, vl);
+
+        vuint16m1_t topRight = __riscv_vmv_v_x_u16m1(aboveblkSize, vl);
+        vuint16m1_t bottomLeft = __riscv_vmv_v_x_u16m1(leftblkSize, vl);
+        vuint16m1_t cx = __riscv_vmv_v_x_u16m1(blkSize - 1, vl);
+        vuint16m1_t oneVec = __riscv_vmv_v_x_u16m1(1, vl);
+        vuint16m1_t blkSizeVec = __riscv_vmv_v_x_u16m1(blkSize, vl);
+
+        // topRight constant
+        vuint16m1_t inc = __riscv_vid_v_u16m1(vl);
+        vuint16m1_t tR0 = __riscv_vadd_vv_u16m1(inc, oneVec, vl);
+
+        // left constant
+        vuint16m1_t l0 = __riscv_vsub_vv_u16m1(blkSizeVec, tR0, vl);
+
+        vuint32m2_t offset = __riscv_vmv_v_x_u32m2(blkSize, vl);
+        vuint32m2_t offset_bottomLeft = __riscv_vadd_vv_u32m2(
+            offset, __riscv_vwcvtu_x_x_v_u32m2(bottomLeft, vl), vl);
+
+        vuint32m2_t t0 = __riscv_vwmaccu_vv_u32m2(offset_bottomLeft, topRight, tR0, vl);
+        t0 = __riscv_vwmaccu_vv_u32m2(t0, above0, cx, vl);
+
+        vuint32m2_t sub_bottomLeft_above0 = __riscv_vwsubu_vv_u32m2(bottomLeft, above0, vl);
+
+        for (int y = 0; y < blkSize; y++) {
+            vuint16m1_t left_v = __riscv_vmv_v_x_u16m1(lefty, vl);
+            vuint32m2_t r0 = __riscv_vwmaccu_vv_u32m2(t0, left_v, l0, vl);
+            vuint16m1_t d0 = __riscv_vnsrl_wx_u16m1(r0, log2Size + 1, vl);
+            __riscv_vse16_v_u16m1(dst + y * dstStride, d0, vl);
+            t0 = __riscv_vadd_vv_u32m2(t0, sub_bottomLeft_above0, vl);
+        }
+    }
+    break;
+    case 16:
+    {
+        size_t vl = 8;
+
+        vuint16m1_t above0 = __riscv_vle16_v_u16m1(above, vl);
+        vuint16m1_t above1 = __riscv_vle16_v_u16m1(above + 8, vl);
+
+        vuint16m1_t topRight = __riscv_vmv_v_x_u16m1(aboveblkSize, vl);
+        vuint16m1_t bottomLeft = __riscv_vmv_v_x_u16m1(leftblkSize, vl);
+
+        vuint16m1_t c15 = __riscv_vmv_v_x_u16m1(15, vl);
+
+        const uint16_t c216 =
+        {
+            {15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0},
+            {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}
+        };
+
+        // left constant
+        vuint16m1_t l0 = __riscv_vle16_v_u16m1(c0, vl);
+        vuint16m1_t l1 = __riscv_vle16_v_u16m1(c0 + 8, vl);
+
+        // topRight constant
+        vuint16m1_t tR0 = __riscv_vle16_v_u16m1(c1, vl);
+        vuint16m1_t tR1 = __riscv_vle16_v_u16m1(c1 + 8, vl);
+
+        vuint32m2_t offset = __riscv_vmv_v_x_u32m2(blkSize, vl);
+        vuint32m2_t offset_bottomLeft = __riscv_vadd_vv_u32m2(
+            offset, __riscv_vwcvtu_x_x_v_u32m2(bottomLeft, vl), vl);
+
+        vuint32m2_t t0 = __riscv_vwmaccu_vv_u32m2(offset_bottomLeft, topRight, tR0, vl);
+        t0 = __riscv_vwmaccu_vv_u32m2(t0, above0, c15, vl);
+
+        vuint32m2_t t1 = __riscv_vwmaccu_vv_u32m2(offset_bottomLeft, topRight, tR1, vl);
+        t1 = __riscv_vwmaccu_vv_u32m2(t1, above1, c15, vl);
+
+        vuint32m2_t sub_bottomLeft_above0 = __riscv_vwsubu_vv_u32m2(bottomLeft, above0, vl);
+        vuint32m2_t sub_bottomLeft_above1 = __riscv_vwsubu_vv_u32m2(bottomLeft, above1, vl);
+
+        for (int y = 0; y < 16; y++) {
+            vuint16m1_t left_v = __riscv_vmv_v_x_u16m1(lefty, vl);
+
+            vuint32m2_t r0 = __riscv_vwmaccu_vv_u32m2(t0, left_v, l0, vl);
+            vuint32m2_t r1 = __riscv_vwmaccu_vv_u32m2(t1, left_v, l1, vl);
+
+            vuint16m1_t d0 = __riscv_vnsrl_wx_u16m1(r0, log2Size + 1, vl);
+            vuint16m1_t d1 = __riscv_vnsrl_wx_u16m1(r1, log2Size + 1, vl);
+
+            __riscv_vse16_v_u16m1(dst + y * dstStride, d0, vl);
+            __riscv_vse16_v_u16m1(dst + 8 + y * dstStride, d1, vl);
+
+            t0 = __riscv_vadd_vv_u32m2(t0, sub_bottomLeft_above0, vl);
+            t1 = __riscv_vadd_vv_u32m2(t1, sub_bottomLeft_above1, vl);
+        }
+    }
+    break;
+    case 32:
+    {
+        // Use rnu rounding mode
+        const unsigned int vxrm = 0;
+
+        size_t vl = blkSize;
+        pixel topRight = aboveblkSize;
+        pixel bottomLeft = leftblkSize;
+
+        vuint32m2_t topRightVec = __riscv_vmv_v_x_u32m2(topRight, vl);
+        vuint32m2_t oneVec = __riscv_vmv_v_x_u32m2(1, vl);
+        vuint32m2_t blkSizeSubOneVec = __riscv_vmv_v_x_u32m2(blkSize - 1, vl);
+
+        for (int y = 0; y < blkSize; y++) {
+            for (int x = 0, inc = 0; x < blkSize; x += vl, inc++) {
+                vl = __riscv_vsetvl_e16m1(blkSize - x);
+                // lefty
+                vuint32m2_t leftYVec = __riscv_vmv_v_x_u32m2(lefty, vl);
+
+                // {0, 1, 2, 3, 4, 5, 6, 7 ...}
+                vuint32m2_t xvec = __riscv_vadd_vv_u32m2(__riscv_vid_v_u32m2(vl), __riscv_vmv_v_x_u32m2(inc * vl, vl), vl);
+
+                // (blkSize - 1 - x) * lefty
+                vuint32m2_t first = __riscv_vsub_vv_u32m2(blkSizeSubOneVec, xvec, vl);
+                first = __riscv_vmul_vv_u32m2(first, leftYVec, vl);
+
+                // (x + 1) * topRight
+                vuint32m2_t second = __riscv_vadd_vv_u32m2(xvec, oneVec, vl);
+                second = __riscv_vmul_vv_u32m2(second, topRightVec, vl);
+
+                vuint32m2_t resVec = __riscv_vadd_vv_u32m2(first, second, vl);
+
+                // (blkSize - 1 - y)
+                vuint32m2_t vlkSizeYVec = __riscv_vmv_v_x_u32m2(blkSize - 1 - y, vl);
+
+                // (blkSize - 1 - y) * abovex
+                vuint32m2_t aboveVec = __riscv_vzext_vf2_u32m2(__riscv_vle16_v_u16m1(above + x, vl), vl);
+                aboveVec = __riscv_vmul_vv_u32m2(aboveVec, vlkSizeYVec, vl);
+
+                resVec = __riscv_vadd_vv_u32m2(resVec, aboveVec, vl);
+                // (y + 1) * bottomLeft
+                vuint32m2_t bottomLeftYVec = __riscv_vmv_v_x_u32m2((y + 1) * bottomLeft, vl);
+                resVec = __riscv_vadd_vv_u32m2(resVec, bottomLeftYVec, vl);
+
+                vuint16m1_t  res =  __riscv_vnclipu_wx_u16m1(resVec, log2Size + 1, vxrm, vl);
+                __riscv_vse16_v_u16m1(dst + y * dstStride + x, res, vl);
+            }
+        }
+    }
+    break;
+    }
+}
+#endif
+
+#if !HIGH_BIT_DEPTH
+void intra_pred_planar32_rvv(pixel *dst, intptr_t dstStride, const pixel *srcPix,
+                              int /*dirMode*/, int /*bFilter*/)
+{
+    const int log2Size = 5;
+    const int blkSize = 1 << log2Size;
+
+    const pixel *src0 = srcPix + 1;
+    const pixel *src1 = srcPix + 2 * blkSize + 1;
+
+    size_t vl = 16;
+
+    vuint8m1_t above0 = __riscv_vle8_v_u8m1(src0, vl);
+    vuint8m1_t above1 = __riscv_vle8_v_u8m1(src0 + 16, vl);
+
+    vuint8m1_t topRight = __riscv_vmv_v_x_u8m1(src0blkSize, vl);
+    vuint8m1_t bottomLeft = __riscv_vmv_v_x_u8m1(src1blkSize, vl);
+
+    vuint8m1_t c31 = __riscv_vmv_v_x_u8m1(31, vl);
+
+    const uint8_t c232 =
+    {
+        {31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16,
+         15, 14, 13, 12, 11, 10,  9,  8,  7,  6,  5,  4,  3,  2,  1,  0},
+        {1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
+         17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32}
+    };
+
+    // left constant
+    vuint8m1_t l0 = __riscv_vle8_v_u8m1(c0, vl);
+    vuint8m1_t l1 = __riscv_vle8_v_u8m1(c0 + 16, vl);
+
+    // topRight constant
+    vuint8m1_t tR0 = __riscv_vle8_v_u8m1(c1, vl);
+    vuint8m1_t tR1 = __riscv_vle8_v_u8m1(c1 + 16 , vl);
+
+    vuint16m2_t offset = __riscv_vmv_v_x_u16m2(blkSize, vl);
+    vuint16m2_t offset_bottomLeft = __riscv_vadd_vv_u16m2(
+        offset, __riscv_vwcvtu_x_x_v_u16m2(bottomLeft, vl), vl);
+
+    vuint16m2_t t0 = __riscv_vwmaccu_vv_u16m2(offset_bottomLeft, topRight, tR0, vl);
+    t0 = __riscv_vwmaccu_vv_u16m2(t0, above0, c31, vl);
+
+    vuint16m2_t t1 = __riscv_vwmaccu_vv_u16m2(offset_bottomLeft, topRight, tR1, vl);
+    t1 = __riscv_vwmaccu_vv_u16m2(t1, above1, c31, vl);
+
+    vuint16m2_t sub_bottomLeft_above0 = __riscv_vwsubu_vv_u16m2(bottomLeft, above0, vl);
+    vuint16m2_t sub_bottomLeft_above1 = __riscv_vwsubu_vv_u16m2(bottomLeft, above1, vl);
+
+    for (int y = 0; y < 32; y++)
+    {
+        vuint8m1_t left = __riscv_vmv_v_x_u8m1(src1y, vl);
+
+        vuint16m2_t r0 = __riscv_vwmaccu_vv_u16m2(t0, left, l0, vl);
+        vuint16m2_t r1 = __riscv_vwmaccu_vv_u16m2(t1, left, l1, vl);
+
+        vuint8m1_t d0 = __riscv_vnsrl_wx_u8m1(r0, log2Size + 1, vl);
+        vuint8m1_t d1 = __riscv_vnsrl_wx_u8m1(r1, log2Size + 1, vl);
+
+        __riscv_vse8_v_u8m1(dst + 0 * 8 + y * dstStride, d0, vl);
+        __riscv_vse8_v_u8m1(dst + 2 * 8 + y * dstStride, d1, vl);
+
+        t0 = __riscv_vadd_vv_u16m2(t0, sub_bottomLeft_above0, vl);
+        t1 = __riscv_vadd_vv_u16m2(t1, sub_bottomLeft_above1, vl);
+    }
+}
+#endif
+
+static void dcPredFilter_rvv(const pixel* above, const pixel* left, pixel* dst, intptr_t dststride, int size)
+{
+    pixel topLeft = (pixel)((above0 + left0 + 2 * dst0 + 2) >> 2);
+    pixel * pdst = dst;
+
+    // Use rnu rounding mode
+    const unsigned int vxrm = 0;
+    size_t vl = size;
+
+    switch (size) {
+    case 32:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint8m2_t vconst_3 = __riscv_vmv_v_x_u8m2(3, vl);
+        vuint8m2_t vabo = __riscv_vle8_v_u8m2(above, vl);
+        vuint8m2_t vdst = __riscv_vle8_v_u8m2(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint16m4_t vdst_ext = __riscv_vwmulu_vv_u16m4(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u16m4(vdst_ext, vabo, vl);
+
+        vuint8m2_t res = __riscv_vnclipu_wx_u8m2(vdst_ext, 2, vxrm, vl);
+        __riscv_vse8_v_u8m2(dst, res, vl);
+    #else
+        vuint16m4_t vconst_3 = __riscv_vmv_v_x_u16m4(3, vl);
+        vuint16m4_t vabo = __riscv_vle16_v_u16m4(above, vl);
+        vuint16m4_t vdst = __riscv_vle16_v_u16m4(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint32m8_t vdst_ext = __riscv_vwmulu_vv_u32m8(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u32m8(vdst_ext, vabo, vl);
+
+        vuint16m4_t res = __riscv_vnclipu_wx_u16m4(vdst_ext, 2, vxrm, vl);
+        __riscv_vse16_v_u16m4(dst, res, vl);
+    #endif
+    }
+    break;
+    case 16:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint8m1_t vconst_3 = __riscv_vmv_v_x_u8m1(3, vl);
+        vuint8m1_t vabo = __riscv_vle8_v_u8m1(above, vl);
+        vuint8m1_t vdst = __riscv_vle8_v_u8m1(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint16m2_t vdst_ext = __riscv_vwmulu_vv_u16m2(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u16m2(vdst_ext, vabo, vl);
+
+        vuint8m1_t res = __riscv_vnclipu_wx_u8m1(vdst_ext, 2, vxrm, vl);
+        __riscv_vse8_v_u8m1(dst, res, vl);
+    #else
+        vuint16m2_t vconst_3 = __riscv_vmv_v_x_u16m2(3, vl);
+        vuint16m2_t vabo = __riscv_vle16_v_u16m2(above, vl);
+        vuint16m2_t vdst = __riscv_vle16_v_u16m2(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint32m4_t vdst_ext = __riscv_vwmulu_vv_u32m4(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u32m4(vdst_ext, vabo, vl);
+
+        vuint16m2_t res = __riscv_vnclipu_wx_u16m2(vdst_ext, 2, vxrm, vl);
+        __riscv_vse16_v_u16m2(dst, res, vl);
+    #endif
+    }
+    break;
+    case 8:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint8mf2_t vconst_3 = __riscv_vmv_v_x_u8mf2(3, vl);
+        vuint8mf2_t vabo = __riscv_vle8_v_u8mf2(above, vl);
+        vuint8mf2_t vdst = __riscv_vle8_v_u8mf2(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint16m1_t vdst_ext = __riscv_vwmulu_vv_u16m1(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u16m1(vdst_ext, vabo, vl);
+
+        vuint8mf2_t res = __riscv_vnclipu_wx_u8mf2(vdst_ext, 2, vxrm, vl);
+        __riscv_vse8_v_u8mf2(dst, res, vl);
+    #else
+        vuint16m1_t vabo = __riscv_vle16_v_u16m1(above, vl);
+        vuint16m1_t vdst = __riscv_vle16_v_u16m1(dst, vl);
+
+        vuint16m1_t vconst_3 = __riscv_vmv_v_x_u16m1(3, vl);
+        vuint16m1_t vconst_2 = __riscv_vmv_v_x_u16m1(2, vl);
+        vuint16m1_t tmp = __riscv_vadd_vv_u16m1(vabo, vconst_2, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint32m2_t vdst_ext = __riscv_vwmulu_vv_u32m2(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u32m2(vdst_ext, tmp, vl);
+
+        vuint16m1_t res = __riscv_vnsrl_wx_u16m1(vdst_ext, 2, vl);
+        __riscv_vse16_v_u16m1(dst, res, vl);
+    #endif
+    }
+    break;
+    case 4:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint8mf4_t vconst_3 = __riscv_vmv_v_x_u8mf4(3, vl);
+        vuint8mf4_t vabo = __riscv_vle8_v_u8mf4(above, vl);
+        vuint8mf4_t vdst = __riscv_vle8_v_u8mf4(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint16mf2_t vdst_ext = __riscv_vwmulu_vv_u16mf2(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u16mf2(vdst_ext, vabo, vl);
+
+        vuint8mf4_t res = __riscv_vnclipu_wx_u8mf4(vdst_ext, 2, vxrm, vl);
+        __riscv_vse8_v_u8mf4(dst, res, vl);
+    #else
+        vuint16mf2_t vconst_3 = __riscv_vmv_v_x_u16mf2(3, vl);
+        vuint16mf2_t vabo = __riscv_vle16_v_u16mf2(above, vl);
+        vuint16mf2_t vdst = __riscv_vle16_v_u16mf2(dst, vl);
+
+        // Compute: dstx = (abovex + 3 * dstx + 2) >> 2
+        vuint32m1_t vdst_ext = __riscv_vwmulu_vv_u32m1(vdst, vconst_3, vl);
+        vdst_ext = __riscv_vwaddu_wv_u32m1(vdst_ext, vabo, vl);
+
+        vuint16mf2_t res = __riscv_vnclipu_wx_u16mf2(vdst_ext, 2, vxrm, vl);
+        __riscv_vse16_v_u16mf2(dst, res, vl);
+    #endif
+    }
+    break;
+    }
+    dst += dststride;
+    for (int y = 1; y < size; y++) {
+        *dst = (pixel)((lefty + 3 * *dst + 2) >> 2);
+        dst += dststride;
+    }
+    *pdst = topLeft;
+}
+
+template<int width>
+void intra_pred_dc_rvv(pixel* dst, intptr_t dstStride, const pixel* srcPix, int /*dirMode*/, int bFilter)
+{
+    int dcVal = width;
+    size_t vl = width;
+
+    switch (width) {
+    case 32:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint16m8_t vsp = __riscv_vmv_v_x_u16m8(0, vl);
+        vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, 1);
+
+        vuint8m4_t spa = __riscv_vle8_v_u8m4(srcPix + 1, vl);
+        vuint8m4_t spb = __riscv_vle8_v_u8m4(srcPix + 2 * width + 1, vl);
+        vuint16m8_t tmp = __riscv_vwaddu_vv_u16m8(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u16m8(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u16m8_u16m1(vsp, v_sum, width);
+        dcVal += __riscv_vmv_x_s_u16m1_u16(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint8m4_t vdc = __riscv_vmv_v_x_u8m4(dcVal, vl);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+             __riscv_vse8_v_u8m4(row, vdc, vl);
+        }
+    #else
+        vuint32m8_t vsp = __riscv_vmv_v_x_u32m8(0, vl);
+        vuint32m1_t v_sum = __riscv_vmv_v_x_u32m1(0, 1);
+
+        vuint16m4_t spa = __riscv_vle16_v_u16m4(srcPix + 1, vl);
+        vuint16m4_t spb = __riscv_vle16_v_u16m4(srcPix + 2 * width + 1, vl);
+        vuint32m8_t tmp = __riscv_vwaddu_vv_u32m8(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u32m8(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u32m8_u32m1(vsp, v_sum, width);
+        dcVal += __riscv_vmv_x_s_u32m1_u32(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint16m4_t vdc = __riscv_vmv_v_x_u16m4(dcVal, vl);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse16_v_u16m4(row, vdc, vl);
+        }
+    #endif
+    }
+    break;
+    case 16:
+    {
+    #if !HIGH_BIT_DEPTH
+
+        vuint16m4_t vsp = __riscv_vmv_v_x_u16m4(0, vl);
+        vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, 1);
+
+        vuint8m2_t spa = __riscv_vle8_v_u8m2(srcPix + 1, vl);
+        vuint8m2_t spb = __riscv_vle8_v_u8m2(srcPix + 2 * width + 1, vl);
+        vuint16m4_t tmp = __riscv_vwaddu_vv_u16m4(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u16m4(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u16m4_u16m1(vsp, v_sum, width);
+        dcVal += __riscv_vmv_x_s_u16m1_u16(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint8m2_t vdc = __riscv_vmv_v_x_u8m2(dcVal, vl);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse8_v_u8m2(row, vdc, vl);
+        }
+    #else
+        vuint32m4_t vsp = __riscv_vmv_v_x_u32m4(0, vl);
+        vuint32m1_t v_sum = __riscv_vmv_v_x_u32m1(0, 1);
+
+        vuint16m2_t spa = __riscv_vle16_v_u16m2(srcPix + 1, vl);
+        vuint16m2_t spb = __riscv_vle16_v_u16m2(srcPix + 2 * width + 1, vl);
+        vuint32m4_t tmp = __riscv_vwaddu_vv_u32m4(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u32m4(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u32m4_u32m1(vsp, v_sum, width);
+        dcVal += __riscv_vmv_x_s_u32m1_u32(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint16m2_t vdc = __riscv_vmv_v_x_u16m2(dcVal, vl);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse16_v_u16m2(row, vdc, vl);
+        }
+    #endif
+    }
+    break;
+    case 8:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint16m1_t vsp = __riscv_vmv_v_x_u16m1(0, vl);
+        vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, 1);
+
+        vuint8mf2_t spa = __riscv_vle8_v_u8mf2(srcPix + 1, vl);
+        vuint8mf2_t spb = __riscv_vle8_v_u8mf2(srcPix + 2 * width + 1, vl);
+        vuint16m1_t tmp = __riscv_vwaddu_vv_u16m1(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u16m1(vsp, tmp, width);
+
+        v_sum = __riscv_vredsum_vs_u16m1_u16m1(vsp, v_sum, width);
+        dcVal += __riscv_vmv_x_s_u16m1_u16(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint8mf2_t vdc = __riscv_vmv_v_x_u8mf2(dcVal, width);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse8_v_u8mf2(row, vdc, width);
+    }
+    #else
+        vuint16m1_t spa = __riscv_vle16_v_u16m1(srcPix + 1, vl);
+        vuint16m1_t spb = __riscv_vle16_v_u16m1(srcPix + 2 * width + 1, vl);
+
+        vuint32m2_t vsp = __riscv_vmv_v_x_u32m2(0, vl);
+        vuint32m1_t v_sum = __riscv_vmv_v_x_u32m1(0, 1);
+
+        vuint32m2_t tmp = __riscv_vwaddu_vv_u32m2(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u32m2(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u32m2_u32m1(vsp, v_sum, vl);
+        dcVal += __riscv_vmv_x_s_u32m1_u32(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint16m1_t vdc = __riscv_vmv_v_x_u16m1(dcVal, width);
+        for (int k = 0; k < width; k += 2) {
+            __riscv_vse16_v_u16m1(dst + k * dstStride, vdc, width);
+            __riscv_vse16_v_u16m1(dst + (k + 1) * dstStride, vdc, width);
+        }
+#endif
+    }
+    break;
+    case 4:
+    {
+    #if !HIGH_BIT_DEPTH
+        vuint16m1_t vsp = __riscv_vmv_v_x_u16m1(0, vl);
+
+        vuint8mf2_t spa = __riscv_vle8_v_u8mf2(srcPix + 1, vl);
+        vuint8mf2_t spb = __riscv_vle8_v_u8mf2(srcPix + 2 * width + 1, vl);
+        vuint16m1_t tmp = __riscv_vwaddu_vv_u16m1(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u16m1(vsp, tmp, width);
+
+        vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, 1);
+        v_sum = __riscv_vredsum_vs_u16m1_u16m1(vsp, v_sum, width);
+
+        dcVal += __riscv_vmv_x_s_u16m1_u16(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint8mf2_t vdc = __riscv_vmv_v_x_u8mf2(dcVal, width);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse8_v_u8mf2(row, vdc, width);
+        }
+    #else
+        vuint16mf2_t vsp = __riscv_vmv_v_x_u16mf2(0, vl);
+        vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, 1);
+
+        vuint16mf2_t spa = __riscv_vle16_v_u16mf2(srcPix + 1, vl);
+        vuint16mf2_t spb = __riscv_vle16_v_u16mf2(srcPix + 2 * width + 1, vl);
+        vuint16mf2_t tmp = __riscv_vadd_vv_u16mf2(spa, spb, vl);
+        vsp = __riscv_vadd_vv_u16mf2(vsp, tmp, vl);
+
+        v_sum = __riscv_vredsum_vs_u16mf2_u16m1(vsp, v_sum, vl);
+        dcVal += __riscv_vmv_x_s_u16m1_u16(v_sum);
+        dcVal = dcVal / (width + width);
+
+        vuint16mf2_t vdc = __riscv_vmv_v_x_u16mf2(dcVal, width);
+        for (int k = 0; k < width; k++) {
+            pixel *row = dst + k * dstStride;
+            __riscv_vse16_v_u16mf2(row, vdc, width);
+        }
+    #endif
+    }
+    break;
+    }
+
+    if (bFilter){
+        dcPredFilter_rvv(srcPix + 1, srcPix + (2 * width + 1), dst, dstStride, width);
+    }
+}
+
+// x265 private namespace
+void setupIntraPrimitives_rvv(EncoderPrimitives &p)
+{
+    p.cuBLOCK_4x4.intra_filter = intraFilter_rvv<4>;
+    p.cuBLOCK_8x8.intra_filter = intraFilter_rvv<8>;
+    p.cuBLOCK_16x16.intra_filter = intraFilter_rvv<16>;
+    p.cuBLOCK_32x32.intra_filter = intraFilter_rvv<32>;
+
+    for (int i = 2; i < NUM_INTRA_MODE; i++) {
+        p.cuBLOCK_4x4.intra_predi = intra_pred_ang_rvv<4>;
+        p.cuBLOCK_8x8.intra_predi = intra_pred_ang_rvv<8>;
+        p.cuBLOCK_16x16.intra_predi = intra_pred_ang_rvv<16>;
+        p.cuBLOCK_32x32.intra_predi = intra_pred_ang_rvv<32>;
+    }
+
+    //p.cuBLOCK_4x4.intra_pred_allangs = all_angs_pred_rvv<2>;
+    p.cuBLOCK_8x8.intra_pred_allangs = all_angs_pred_rvv<3>;
+    p.cuBLOCK_16x16.intra_pred_allangs = all_angs_pred_rvv<4>;
+    p.cuBLOCK_32x32.intra_pred_allangs = all_angs_pred_rvv<5>;
+
+#if !HIGH_BIT_DEPTH
+    p.cuBLOCK_32x32.intra_predPLANAR_IDX = intra_pred_planar32_rvv;
+#else
+    p.cuBLOCK_4x4.intra_predPLANAR_IDX = planar_pred_rvv<2>;
+    p.cuBLOCK_8x8.intra_predPLANAR_IDX = planar_pred_rvv<3>;
+    p.cuBLOCK_16x16.intra_predPLANAR_IDX = planar_pred_rvv<4>;
+    p.cuBLOCK_32x32.intra_predPLANAR_IDX = planar_pred_rvv<5>;
+#endif
+
+    p.cuBLOCK_4x4.intra_predDC_IDX = intra_pred_dc_rvv<4>;
+    p.cuBLOCK_8x8.intra_predDC_IDX = intra_pred_dc_rvv<8>;
+    p.cuBLOCK_16x16.intra_predDC_IDX = intra_pred_dc_rvv<16>;
+    p.cuBLOCK_32x32.intra_predDC_IDX = intra_pred_dc_rvv<32>;
+
+}
+}

x265-4.2.tar/source/common/riscv64/intrapred.S Added

@@ -0,0 +1,270 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+ #include "asm.S"
+
+.section .rodata
+.align 4
+
+tbl_const_1to5_3to0:
+    .byte 1, 2, 3, 4
+    .byte 3, 2, 1, 0
+
+.align 4
+tbl_const_1to8_7to0:
+    .byte 1, 2, 3, 4, 5, 6, 7, 8          // coefficient of (x+1)*TR
+    .byte 7, 6, 5, 4, 3, 2, 1, 0          // coefficient of (blksize-1-x)*Ly
+
+.align 4
+tbl_const_1to16_15to0:
+    .byte 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
+    .byte 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
+
+.section .text
+.align 4
+
+
+function PFX(intra_pred_planar4_rvv)
+    li t0, 4
+    vsetvli t0, t0, e8, mf2, ta, ma
+
+    addi t1, a2, 1
+    vle8.v v0, (t1)           # v0 = above0..3
+
+    lbu t2, 5(a2)
+    vmv.v.x v2, t2            # v2 = topRight_b_broadcast
+
+    lbu t3, 13(a2)
+    vsetvli zero, zero, e16, m1, ta, ma
+    vmv.v.x v3, t3            # v3 = bottomLeft_h_broadcast
+
+    la t2, tbl_const_1to5_3to0
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vle8.v v4, (t2)           # v4 = 1,2,3,4 (x+1)
+    addi t2, t2, 4
+    vle8.v v5, (t2)           # v5 = 3,2,1,0 (3-x)
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vzext.vf2 v6, v0          # zero extension to 16 bits
+    vsll.vi v6, v6, 2         # v6 = 4 * abovex
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vwmulu.vv v10, v4, v2      # v10 = (x+1) * topRight
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v6, v6, v10        # v6 = 4 * abovex + (x+1) * topRight
+
+    vzext.vf2 v12, v0          # v12 = above (zero extension to 16 bits)
+    vsub.vv v8, v3, v12        # v8 = bottomLeft - abovex, as a correction
+
+    csrwi  vxrm, 0
+    li t4, 0                  # y = 0
+    mv t5, a0                 # t5 = current dst
+    mv t6, a1                 # t6 = dstStride
+
+.rept 4
+    add t1, a2, t4
+    addi t1, t1, 9           # left = srcPix + 9 + y
+    lbu t2, 0(t1)             # t2 = lefty
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vmv.v.x v9, t2            # v9 = lefty_broadcast
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v6, v6, v8        # v6 = (blkSize - 1 -y) * abovex + (x+1) * topRight + (y+1) * bottomLeft
+
+    vmv.v.v v16, v6           # v16 = v6
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vwmulu.vv v14, v5, v9                  # v14 = (3-x) * lefty
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v16, v16, v14                  # v16 += (3-x) * lefty
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vnclipu.wi v18, v16, 3
+
+    vse8.v v18, (t5)
+
+    add     t5, t5, t6          # dst += dstStride
+    addi    t4, t4, 1           # y++
+.endr
+    ret
+endfunc
+
+
+function PFX(intra_pred_planar8_rvv)
+// Register map
+// a0  = dst
+// a1  = dstStride
+// a2  = *srcPix
+// a3  = leftx
+// a4  = tmp
+// v0  = above7:0
+// v1  = left7:0
+// v2  = topRight = rep(aboveblkSize)
+// v3  = bottomLeft = rep(leftblkSize)
+// v4  = const1 2 3 4 5 6 7 8
+// v5  = const7 6 5 4 3 2 1 0
+
+    li t0, 8
+    vsetvli t0, t0, e8, mf2, ta, ma
+
+    addi t1, a2, 1
+    vle8.v v0, (t1)           # v0 = above0..7
+
+    lbu t2, 9(a2)
+    vmv.v.x v2, t2            # v2 = topRight_b_broadcast
+
+    lbu t3, 25(a2)
+    vsetvli zero, zero, e16, m1, ta, ma
+    vmv.v.x v3, t3            # v3 = bottomLeft_h_broadcast
+
+    la t2, tbl_const_1to8_7to0
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vle8.v v4, (t2)           # v4 = 1,2,3,4,5,6,7,8 (x+1)
+    addi t2, t2, 8
+    vle8.v v5, (t2)           # v5 = 7,6,5,4,3,2,1,0 (7-x)
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vzext.vf2 v6, v0          # zero extension to 16 bits
+    vsll.vi v6, v6, 3         # v6 = 8 * abovex
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vwmulu.vv v10, v4, v2      # v10 = (x+1) * topRight
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v6, v6, v10        # v6 = 8 * abovex + (x+1) * topRight
+
+    vzext.vf2 v12, v0          # v12 = above (zero extension to 16 bits)
+    vsub.vv v8, v3, v12        # v8 = bottomLeft - abovex, as a correction
+
+    csrwi  vxrm, 0
+    li t3, 8
+    li t4, 0                  # y = 0
+    mv t5, a0                 # t5 = current dst
+    mv t6, a1                 # t6 = dstStride
+
+loop_y_8x8:
+    add t1, a2, t4
+    addi t1, t1, 17           # left = srcPix + 17 + y
+    lbu t2, 0(t1)             # t2 = lefty
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vmv.v.x v9, t2            # v9 = lefty_broadcast
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v6, v6, v8        # v6 = (blkSize - 1 -y) * abovex + (x+1) * topRight + (y+1) * bottomLeft
+
+    vmv.v.v v16, v6           # v16 = v6
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vwmulu.vv v14, v5, v9                  # v14 = (7-x) * lefty
+
+    vsetvli zero, zero, e16, m1, ta, ma
+    vadd.vv v16, v16, v14                  # v16 += (7-x) * lefty
+
+    vsetvli zero, zero, e8, mf2, ta, ma
+    vnclipu.wi v18, v16, 4
+
+    vse8.v v18, (t5)
+
+    add     t5, t5, t6          # dst += dstStride
+    addi    t4, t4, 1           # y++
+    addi    t3, t3, -1
+    bnez    t3, loop_y_8x8
+
+    ret
+endfunc
+
+
+function PFX(intra_pred_planar16_rvv)
+    li t0, 16
+    vsetvli t0, t0, e8, m1, ta, ma
+
+    addi t1, a2, 1
+    vle8.v v0, (t1)           # v0 = above0..15
+
+    lbu t2, 17(a2)
+    vmv.v.x v2, t2            # topRight_b_broadcast
+
+    lbu t3, 49(a2)
+    vsetvli zero, zero, e16, m2, ta, ma
+    vmv.v.x v4, t3            # {v4, v5} = bottomLeft_h_broadcast
+
+    la t2, tbl_const_1to16_15to0
+    vsetvli zero, zero, e8, m1, ta, ma
+    vle8.v v6, (t2)           # v6 = 1,2,3,...,16 (x+1)
+    addi t2, t2, 16
+    vle8.v v7, (t2)           # v7 = 15,14,13,...,0 (15-x)
+
+    vsetvli zero, zero, e16, m2, ta, ma
+    vzext.vf2 v8, v0
+    vsll.vi v8, v8, 4         # v8 = 16 * abovex
+
+    vsetvli zero, zero, e8, m1, ta, ma
+    vwmulu.vv v10, v6, v2     # v10 = (x+1) * topRight
+
+    vsetvli zero, zero, e16, m2, ta, ma
+    vadd.vv v8, v8, v10       # {v8, v9} = 16 * abovex + (x+1) * topRight
+
+    vzext.vf2 v24, v0
+    vsub.vv v12, v4, v24      # {v12, v13} = bottomLeft - abovex
+
+    csrwi  vxrm, 0
+    li t3, 16
+    li t4, 0
+    mv t5, a0
+    mv t6, a1
+
+loop_y_16x16:
+    add t1, a2, t4
+    addi t1, t1, 33           # left = srcPix + 33 + y
+    lbu t2, 0(t1)             # t2 = lefty
+
+    vsetvli zero, zero, e8, m1, ta, ma
+    vmv.v.x v14, t2           # v14 = lefty_broadcast
+
+    vsetvli zero, zero, e16, m2, ta, ma
+    vadd.vv v8, v8, v12       # {v8, v9} +=  = (blkSize - 1 -y) * abovex + (x+1) * topRight + (y+1) * bottomLeft
+
+    vmv.v.v v16, v8           # v16 = v8
+
+    vsetvli zero, zero, e8, m1, ta, ma
+    vwmulu.vv v18, v7, v14    # v18 = (15-x) * lefty
+
+    vsetvli zero, zero, e16, m2, ta, ma
+    vadd.vv v16, v16, v18     # v16 += (15-x) * lefty
+
+    vsetvli zero, zero, e8, m1, ta, ma
+    vnclipu.wi v20, v16, 5
+
+    vse8.v v20, (t5)
+
+    add     t5, t5, t6          # dst += dstStride
+    addi    t4, t4, 1           # y++
+    addi    t3, t3, -1
+    bnez    t3, loop_y_16x16
+
+    ret
+endfunc

x265-4.2.tar/source/common/riscv64/loopfilter.S Added

@@ -0,0 +1,889 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+pelFilterLumaStrong_coeff_16:
+    .hword 0, 2, 1, 1, 2, 3, 3, 7
+    .hword 100, 4, 2, 5, 6, 4, 4, 100
+    .hword 100, 1, 3, 2, 3, 5, 5, 100
+    .hword 100, 0, 4, 3, 4, 6, 6, 100
+    .hword 100, 3, 100, 4, 5, 100, 7, 100
+
+    .hword 0, 3, 1, 2, 2, 1, 1, 0
+    .hword 0, 2, 1, 2, 2, 1, 3, 0
+    .hword 0, 1, 1, 2, 2, 1, 2, 0
+
+    .hword 0, 4, 2, 4, 4, 2, 4, 0
+    .hword 0, 3, 2, 3, 3, 2, 3, 0
+
+.align 4
+.text
+
+//void processSaoCUE0(pixel * rec, int8_t * offsetEo, int width, int8_t* signLeft, intptr_t stride)
+function PFX(processSaoCUE0_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a4, a4, 1
+#endif
+    li              a5, (1 << BIT_DEPTH) - 1
+    lb              t0, (a3)
+    lb              t1, 1(a3)
+    neg             t0, t0
+    neg             t1, t1
+    add             t3, a0, a4
+    vsetvli         zero, a2, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v28, a5
+    vsetvli         zero, a2, e8, m1, ta, ma
+    vmv.s.x         v8, t0
+    vmv.s.x         v9, t1
+loop_processSaoCUE0:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a2, e16, m2, ta, ma
+    addi            t4, a0, 2
+    addi            t5, t3, 2
+    vle16.v         v2, (a0)
+    vle16.v         v20, (t4)
+    vle16.v         v4, (t3)
+    vle16.v         v22, (t5)
+    SIGNOF_RVV      v24, v2, v20
+    SIGNOF_RVV      v26, v4, v22
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v6, v24, 0
+    vnsrl.wi        v7, v26, 0
+#else
+    vsetvli         t2, a2, e8, m1, ta, ma
+    addi            t4, a0, 1
+    addi            t5, t3, 1
+    vle8.v          v2, (a0)
+    vle8.v          v20, (t4)
+    vle8.v          v4, (t3)
+    vle8.v          v22, (t5)
+    SIGNOF_RVV      v6, v2, v20
+    SIGNOF_RVV      v7, v4, v22
+#endif
+    addi            t6, t2, -1
+    vslideup.vi     v8, v6, 1
+    vslideup.vi     v9, v7, 1
+    vsub.vv         v10, v6, v8
+    vsub.vv         v11, v7, v9
+    vadd.vi         v10, v10, 2
+    vadd.vi         v11, v11, 2
+    vslidedown.vx   v8, v6, t6
+    vslidedown.vx   v9, v7, t6
+
+#if HIGH_BIT_DEPTH
+    vluxei8.v       v20, (a1), v10
+    vluxei8.v       v22, (a1), v11
+    vwadd.wv        v2, v2, v20
+    vwadd.wv        v4, v4, v22
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v2, v2, v30
+    vmax.vv         v4, v4, v30
+    vmin.vv         v2, v2, v28
+    vmin.vv         v4, v4, v28
+    vse16.v         v2, (a0)
+    vse16.v         v4, (t3)
+#else
+    vluxei8.v       v20, (a1), v10
+    vluxei8.v       v22, (a1), v11
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v12, v2
+    vzext.vf2       v14, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vwadd.wv        v12, v12, v20
+    vwadd.wv        v14, v14, v22
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v12, v12, v30
+    vmax.vv         v14, v14, v30
+    vmin.vv         v12, v12, v28
+    vmin.vv         v14, v14, v28
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v16, v12, 0
+    vnsrl.wi        v17, v14, 0
+    vse8.v          v16, (a0)
+    vse8.v          v17, (t3)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t1, t2, 1
+    add             a0, a0, t1
+    add             t3, t3, t1
+#else
+    add             a0, a0, t2
+    add             t3, t3, t2
+#endif
+    sub             a2, a2, t2
+    bgtz            a2, loop_processSaoCUE0
+    ret
+endfunc
+
+//void processSaoCUE1(pixel* rec, int8_t* upBuff1, int8_t* offsetEo, intptr_t stride, int width)
+function PFX(processSaoCUE1_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+#endif
+    li              a5, (1 << BIT_DEPTH) - 1
+    add             t0, a0, a3
+    vsetvli         zero, a4, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v28, a5
+loop_processSaoCUE1:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a4, e16, m2, ta, ma
+    vle16.v         v14, (a0)
+    vle16.v         v4, (t0)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v20, v14, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v8, v20, 0
+#else
+    vsetvli         t2, a4, e8, m1, ta, ma
+    vle8.v          v2, (a0)
+    vle8.v          v4, (t0)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v8, v2, v4
+#endif
+    vadd.vv         v10, v8, v6
+    vadd.vi         v10, v10, 2
+    vrsub.vi        v8, v8, 0
+    vse8.v          v8, (a1)
+    vluxei8.v       v12, (a2), v10
+
+#if !HIGH_BIT_DEPTH
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v14, v2
+    vsetvli         zero, t2, e8, m1, ta, ma
+#endif
+    vwadd.wv        v14, v14, v12
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v14, v14, v30
+    vmin.vv         v14, v14, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v14, (a0)
+#else
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v16, v14, 0
+    vse8.v          v16, (a0)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t3, t2, 1
+    add             a0, a0, t3
+    add             t0, t0, t3
+#else
+    add             a0, a0, t2
+    add             t0, t0, t2
+#endif
+    add             a1, a1, t2
+    sub             a4, a4, t2
+    bgtz            a4, loop_processSaoCUE1
+    ret
+endfunc
+
+//void processSaoCUE1_2Rows(pixel* rec, int8_t* upBuff1, int8_t* offsetEo, intptr_t stride, int width)
+function PFX(processSaoCUE1_2Rows_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+#endif
+    li              a5, (1 << BIT_DEPTH) - 1
+    add             t0, a0, a3
+    add             t4, t0, a3
+    vsetvli         zero, a4, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v18, a5
+loop_processSaoCUE1_2Rows:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a4, e16, m2, ta, ma
+    vle16.v         v14, (a0)
+    vle16.v         v28, (t0)
+    vle16.v         v2, (t4)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v20, v14, v28
+    SIGNOF_RVV      v4, v28, v2
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v8, v20, 0
+    vnsrl.wi        v22, v4, 0
+#else
+    vsetvli         t2, a4, e8, m1, ta, ma
+    vle8.v          v2, (a0)
+    vle8.v          v4, (t0)
+    vle8.v          v20, (t4)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v8, v2, v4
+    SIGNOF_RVV      v22, v4, v20
+#endif
+    vadd.vv         v10, v8, v6
+    vadd.vi         v10, v10, 2
+    vrsub.vi        v8, v8, 0
+    vadd.vv         v24, v22, v8
+    vadd.vi         v24, v24, 2
+    vrsub.vi        v8, v22, 0
+    vse8.v          v8, (a1)
+    vluxei8.v       v12, (a2), v10
+    vluxei8.v       v26, (a2), v24
+
+#if !HIGH_BIT_DEPTH
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v14, v2
+    vzext.vf2       v28, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+#endif
+    vwadd.wv        v14, v14, v12
+    vwadd.wv        v28, v28, v26
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v14, v14, v30
+    vmax.vv         v28, v28, v30
+    vmin.vv         v14, v14, v18
+    vmin.vv         v28, v28, v18
+#if HIGH_BIT_DEPTH
+    vse16.v         v14, (a0)
+    vse16.v         v28, (t0)
+#else
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v16, v14, 0
+    vnsrl.wi        v12, v28, 0
+    vse8.v          v16, (a0)
+    vse8.v          v12, (t0)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t3, t2, 1
+    add             a0, a0, t3
+    add             t0, t0, t3
+    add             t4, t4, t3
+#else
+    add             a0, a0, t2
+    add             t0, t0, t2
+    add             t4, t4, t2
+#endif
+    add             a1, a1, t2
+    sub             a4, a4, t2
+    bgtz            a4, loop_processSaoCUE1_2Rows
+    ret
+endfunc
+
+//void processSaoCUE2(pixel * rec, int8_t * bufft, int8_t * buff1, int8_t * offsetEo, int width, intptr_t stride)
+function PFX(processSaoCUE2_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a5, a5, 1
+    add             t0, a0, a5
+    addi            t0, t0, 2
+#else
+    add             t0, a0, a5
+    addi            t0, t0, 1
+#endif
+    li              a6, (1 << BIT_DEPTH) - 1
+    addi            a1, a1, 1
+    vsetvli         zero, a4, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v28, a6
+
+loo_processSaoCUE2:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a4, e16, m2, ta, ma
+    vle16.v         v16, (a0)
+    vle16.v         v4, (t0)
+    vle8.v          v6, (a2)
+    SIGNOF_RVV      v10, v16, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v8, v10, 0
+#else
+    vsetvli         t2, a4, e8, m1, ta, ma
+    vle8.v          v2, (a0)
+    vle8.v          v4, (t0)
+    vle8.v          v6, (a2)
+    SIGNOF_RVV      v8, v2, v4
+#endif
+    vadd.vv         v12, v8, v6
+    vadd.vi         v12, v12, 2
+    vrsub.vi        v8, v8, 0
+    vse8.v          v8, (a1)
+    vluxei8.v       v14, (a3), v12
+
+#if !HIGH_BIT_DEPTH
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v16, v2
+    vsetvli         zero, t2, e8, m1, ta, ma
+#endif
+    vwadd.wv        v16, v16, v14
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v16, v16, v30
+    vmin.vv         v16, v16, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v16, (a0)
+#else
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v18, v16, 0
+    vse8.v          v18, (a0)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t3, t2, 1
+    add             a0, a0, t3
+    add             t0, t0, t3
+#else
+    add             a0, a0, t2
+    add             t0, t0, t2
+#endif
+    add             a1, a1, t2
+    add             a2, a2, t2
+    sub             a4, a4, t2
+    bgtz            a4, loo_processSaoCUE2
+    ret
+endfunc
+
+//void processSaoCUE3(pixel *rec, int8_t *upBuff1, int8_t *offsetEo, intptr_t stride, int startX, int endX)
+function PFX(processSaoCUE3_rvv)
+#if HIGH_BIT_DEPTH
+    slli            t1, a4, 1
+    add             a0, a0, t1
+    addi            a0, a0, 2
+    slli            a3, a3, 1
+    add             t0, a0, a3
+#else
+    add             a0, a0, a4
+    addi            a0, a0, 1
+    add             t0, a0, a3
+#endif
+    add             a1, a1, a4
+    addi            a1, a1, 1
+    sub             a4, a5, a4
+    addi            a4, a4, -1
+    addi            t5, a1, -1
+    bltz            a4, processSaoCUE3_finish
+    li              a6, (1 << BIT_DEPTH) - 1
+    vsetvli         zero, a4, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v28, a6
+
+loo_processSaoCUE3:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a4, e16, m2, ta, ma
+    vle16.v         v16, (a0)
+    vle16.v         v4, (t0)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v10, v16, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v8, v10, 0
+#else
+    vsetvli         t2, a4, e8, m1, ta, ma
+    vle8.v          v2, (a0)
+    vle8.v          v4, (t0)
+    vle8.v          v6, (a1)
+    SIGNOF_RVV      v8, v2, v4
+#endif
+    vadd.vv         v12, v8, v6
+    vadd.vi         v12, v12, 2
+    vrsub.vi        v8, v8, 0
+    vse8.v          v8, (t5)
+    vluxei8.v       v14, (a2), v12
+
+#if !HIGH_BIT_DEPTH
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v16, v2
+    vsetvli         zero, t2, e8, m1, ta, ma
+#endif
+    vwadd.wv        v16, v16, v14
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v16, v16, v30
+    vmin.vv         v16, v16, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v16, (a0)
+#else
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v18, v16, 0
+    vse8.v          v18, (a0)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t3, t2, 1
+    add             a0, a0, t3
+    add             t0, t0, t3
+#else
+    add             a0, a0, t2
+    add             t0, t0, t2
+#endif
+    add             t5, t5, t2
+    add             a1, a1, t2
+    sub             a4, a4, t2
+    bgtz            a4, loo_processSaoCUE3
+processSaoCUE3_finish:
+    ret
+endfunc
+
+//void processSaoCUB0(pixel* rec, const int8_t* offset, int ctuWidth, int ctuHeight, intptr_t stride)
+function PFX(processSaoCUB0_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a4, a4, 1
+#endif
+    li              a6, (1 << BIT_DEPTH) - 1
+    vsetvli         zero, a2, e16, m2, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v28, a6
+loop_processSaoCUB0_y:
+    mv              t0, a0
+    mv              t1, a2
+loop_processSaoCUB0_x:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, t1, e16, m2, ta, ma
+    vle16.v         v16, (t0)
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v4, v16, BIT_DEPTH - 5
+#else
+    vsetvli         t2, t1, e8, m1, ta, ma
+    vle8.v          v2, (t0)
+    vsrl.vi         v4, v2, BIT_DEPTH - 5
+#endif
+    vluxei8.v       v6, (a1), v4
+
+#if !HIGH_BIT_DEPTH
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vzext.vf2       v16, v2
+    vsetvli         zero, t2, e8, m1, ta, ma
+#endif
+    vwadd.wv        v16, v16, v6
+    vsetvli         zero, t2, e16, m2, ta, ma
+    vmax.vv         v16, v16, v30
+    vmin.vv         v16, v16, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v16, (t0)
+#else
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v18, v16, 0
+    vse8.v          v18, (t0)
+#endif
+
+#if HIGH_BIT_DEPTH
+    slli            t3, t2, 1
+    add             t0, t0, t3
+#else
+    add             t0, t0, t2
+#endif
+    sub             t1, t1, t2
+    bgtz            t1, loop_processSaoCUB0_x
+    addi            a3, a3, -1
+    add             a0, a0, a4
+    bgtz            a3, loop_processSaoCUB0_y
+    ret
+endfunc
+
+//void calSign(int8_t *dst, const pixel *src1, const pixel *src2, const int endX)
+function PFX(calSign_rvv)
+loop_calSign:
+#if HIGH_BIT_DEPTH
+    vsetvli         t2, a3, e16, m2, ta, ma
+    vle16.v         v2, (a1)
+    vle16.v         v4, (a2)
+    SIGNOF_RVV      v6, v2, v4
+    vsetvli         zero, t2, e8, m1, ta, ma
+    vnsrl.wi        v8, v6, 0
+    vse8.v          v8, (a0)
+    slli            t3, t2, 1
+    add             a0, a0, t2
+    add             a1, a1, t3
+    add             a2, a2, t3
+#else
+    vsetvli         t2, a3, e8, m1, ta, ma
+    vle8.v          v2, (a1)
+    vle8.v          v4, (a2)
+    SIGNOF_RVV      v6, v2, v4
+    vse8.v          v6, (a0)
+    add             a0, a0, t2
+    add             a1, a1, t2
+    add             a2, a2, t2
+#endif
+    sub             a3, a3, t2
+    bgtz            a3, loop_calSign
+    ret
+endfunc
+
+
+//void pelFilterLumaStrong_c(pixel* src, intptr_t srcStep, intptr_t offset, int32_t tcP, int32_t tcQ)
+function PFX(pelFilterLumaStrong_v_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    addi            a0, a0, -8
+#else
+    addi            a0, a0, -4
+#endif
+    slli            a6, a1, 1
+    vsetivli        zero, 8, e16, m1, ta, ma
+    la              t0, pelFilterLumaStrong_coeff_16
+    addi            t1, t0, 16
+    addi            t2, t0, 32
+    addi            t3, t0, 48
+    addi            t4, t0, 64
+    vle16.v         v6, (t0)
+    vle16.v         v7, (t1)
+    vle16.v         v8, (t2)
+    vle16.v         v9, (t3)
+    vle16.v         v10, (t4)
+    addi            t1, t0, 80
+    addi            t2, t0, 96
+    addi            t3, t0, 112
+    addi            t4, t0, 128
+    addi            t5, t0, 144
+    vle16.v         v12, (t1)
+    vle16.v         v13, (t2)
+    vle16.v         v14, (t3)
+    vle16.v         v15, (t4)
+    vle16.v         v16, (t5)
+    vmv.v.x         v4, a3
+    vmv.v.x         v5, a4
+    vslideup.vi     v4, v5, 4
+    vrsub.vi        v5, v4, 0
+
+.rept 2
+    add             a7, a0, a1
+#if HIGH_BIT_DEPTH
+    vle16.v         v22, (a0)
+    vle16.v         v28, (a7)
+#else
+    vsetivli        zero, 8, e8, m1, ta, ma
+    vle8.v          v0, (a0)
+    vle8.v          v1, (a7)
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vzext.vf2       v22, v0
+    vzext.vf2       v28, v1
+#endif
+
+    vrgather.vv     v17, v22, v6
+    vrgather.vv     v18, v22, v7
+    vrgather.vv     v19, v22, v8
+    vrgather.vv     v20, v22, v9
+    vrgather.vv     v21, v22, v10
+    vrgather.vv     v23, v28, v6
+    vrgather.vv     v24, v28, v7
+    vrgather.vv     v25, v28, v8
+    vrgather.vv     v26, v28, v9
+    vrgather.vv     v27, v28, v10
+
+    vmacc.vv        v17, v19, v12
+    vmacc.vv        v23, v25, v12
+    vmacc.vv        v18, v20, v13
+    vmacc.vv        v24, v26, v13
+    vmacc.vv        v17, v21, v14
+    vmacc.vv        v23, v27, v14
+    vadd.vv         v17, v17, v18
+    vadd.vv         v23, v23, v24
+    vadd.vv         v17, v17, v15
+    vadd.vv         v23, v23, v15
+    vsra.vv         v17, v17, v16
+    vsra.vv         v23, v23, v16
+    vsub.vv         v17, v17, v22
+    vsub.vv         v23, v23, v28
+
+    vmin.vv         v17, v17, v4
+    vmin.vv         v23, v23, v4
+    vmax.vv         v17, v17, v5
+    vmax.vv         v23, v23, v5
+
+    vadd.vv         v18, v17, v22
+    vadd.vv         v24, v23, v28
+
+#if HIGH_BIT_DEPTH
+    vse16.v         v18, (a0)
+    vse16.v         v24, (a7)
+#else
+    vsetivli        zero, 8, e8, m1, ta, ma
+    vnsrl.wi        v17, v18, 0
+    vnsrl.wi        v23, v24, 0
+    vse8.v          v17, (a0)
+    vse8.v          v23, (a7)
+    vsetivli        zero, 8, e16, m1, ta, ma
+#endif
+
+    add             a0, a0, a6
+.endr
+    ret
+endfunc
+
+//void pelFilterLumaStrong_c(pixel* src, intptr_t srcStep, intptr_t offset, int32_t tcP, int32_t tcQ)
+function PFX(pelFilterLumaStrong_h_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+#endif
+    slli            t0, a2, 2
+    sub             a0, a0, t0
+    add             t0, a0, a2
+    add             t1, t0, a2
+    add             t2, t1, a2
+    add             t3, t2, a2
+    add             t4, t3, a2
+    add             t5, t4, a2
+    add             t6, t5, a2
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vmv.v.i         v30, 2
+
+#if HIGH_BIT_DEPTH
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vle16.v         v0, (a0)
+    vle16.v         v1, (t0)
+    vle16.v         v2, (t1)
+    vle16.v         v3, (t2)
+    vle16.v         v4, (t3)
+    vle16.v         v5, (t4)
+    vle16.v         v6, (t5)
+    vle16.v         v7, (t6)
+#else
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vle8.v          v8, (a0)
+    vle8.v          v9, (t0)
+    vle8.v          v10, (t1)
+    vle8.v          v11, (t2)
+    vle8.v          v12, (t3)
+    vle8.v          v13, (t4)
+    vle8.v          v14, (t5)
+    vle8.v          v15, (t6)
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vzext.vf2       v0, v8
+    vzext.vf2       v1, v9
+    vzext.vf2       v2, v10
+    vzext.vf2       v3, v11
+    vzext.vf2       v4, v12
+    vzext.vf2       v5, v13
+    vzext.vf2       v6, v14
+    vzext.vf2       v7, v15
+#endif
+    vadd.vv         v8, v3, v4
+    vadd.vv         v22, v1, v5
+    vadd.vv         v20, v0, v1
+    vadd.vv         v9, v8, v2
+    vadd.vv         v10, v8, v5
+    vadd.vv         v28, v2, v6
+    vmacc.vv        v22, v9, v30
+    vadd.vv         v26, v9, v1
+    vadd.vv         v24, v10, v6
+    vadd.vv         v18, v6, v7
+    vmadd.vv        v20, v30, v26
+    vmacc.vv        v28, v10, v30
+    vmadd.vv        v18, v30, v24
+
+    vadd.vi         v20, v20, 4
+    vadd.vi         v26, v26, 2
+    vadd.vi         v22, v22, 4
+    vadd.vi         v28, v28, 4
+    vadd.vi         v24, v24, 2
+    vadd.vi         v18, v18, 4
+    vsra.vi         v20, v20, 3
+    vsra.vi         v26, v26, 2
+    vsra.vi         v22, v22, 3
+    vsra.vi         v28, v28, 3
+    vsra.vi         v24, v24, 2
+    vsra.vi         v18, v18, 3
+    vsub.vv         v20, v20, v1
+    vsub.vv         v26, v26, v2
+    vsub.vv         v22, v22, v3
+    vsub.vv         v28, v28, v4
+    vsub.vv         v24, v24, v5
+    vsub.vv         v18, v18, v6
+
+    vmv.v.x         v10, a3
+    vmv.v.x         v12, a4
+    vrsub.vi        v11, v10, 0
+    vrsub.vi        v13, v12, 0
+    vmax.vv         v20, v20, v11
+    vmax.vv         v26, v26, v11
+    vmax.vv         v22, v22, v11
+    vmax.vv         v28, v28, v13
+    vmax.vv         v24, v24, v13
+    vmax.vv         v18, v18, v13
+    vmin.vv         v20, v20, v10
+    vmin.vv         v26, v26, v10
+    vmin.vv         v22, v22, v10
+    vmin.vv         v28, v28, v12
+    vmin.vv         v24, v24, v12
+    vmin.vv         v18, v18, v12
+    vadd.vv         v20, v20, v1
+    vadd.vv         v26, v26, v2
+    vadd.vv         v22, v22, v3
+    vadd.vv         v28, v28, v4
+    vadd.vv         v24, v24, v5
+    vadd.vv         v18, v18, v6
+
+#if HIGH_BIT_DEPTH
+    vse16.v         v20, (t0)
+    vse16.v         v26, (t1)
+    vse16.v         v22, (t2)
+    vse16.v         v28, (t3)
+    vse16.v         v24, (t4)
+    vse16.v         v18, (t5)
+#else
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vnsrl.wi        v0, v20, 0
+    vnsrl.wi        v1, v26, 0
+    vnsrl.wi        v2, v22, 0
+    vnsrl.wi        v3, v28, 0
+    vnsrl.wi        v4, v24, 0
+    vnsrl.wi        v5, v18, 0
+    vse8.v          v0, (t0)
+    vse8.v          v1, (t1)
+    vse8.v          v2, (t2)
+    vse8.v          v3, (t3)
+    vse8.v          v4, (t4)
+    vse8.v          v5, (t5)
+#endif
+    ret
+endfunc
+
+//void pelFilterChroma_V_c(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc, int32_t maskP, int32_t maskQ)
+function PFX(pelFilterChroma_V_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    addi            a0, a0, -4
+    addi            t0, a0, 2
+    addi            t1, a0, 4
+    addi            t2, a0, 6
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vlse16.v        v2, (a0), a1
+    vlse16.v        v3, (t0), a1
+    vlse16.v        v4, (t1), a1
+    vlse16.v        v5, (t2), a1
+#else
+    addi            a0, a0, -2
+    addi            t0, a0, 1
+    addi            t1, a0, 2
+    addi            t2, a0, 3
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vlse8.v         v6, (a0), a1
+    vlse8.v         v7, (t0), a1
+    vlse8.v         v8, (t1), a1
+    vlse8.v         v9, (t2), a1
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vzext.vf2       v2, v6
+    vzext.vf2       v3, v7
+    vzext.vf2       v4, v8
+    vzext.vf2       v5, v9
+#endif
+    li              a6, (1 << BIT_DEPTH) - 1
+    neg             t3, a3
+    vmv.v.i         v27, 4
+    vmv.v.i         v25, 0
+    vmv.v.x         v28, a3
+    vmv.v.x         v29, t3
+    vmv.v.x         v30, a4
+    vmv.v.x         v31, a5
+    vmv.v.x         v26, a6
+    vsub.vv         v6, v4, v3
+    vsub.vv         v7, v2, v5
+    vmacc.vv        v7, v6, v27
+    vadd.vi         v7, v7, 4
+    vsra.vi         v7, v7, 3
+    vmax.vv         v7, v7, v29
+    vmin.vv         v7, v7, v28
+
+    vand.vv         v8, v7, v30
+    vand.vv         v10, v7, v31
+    vadd.vv         v8, v8, v3
+    vsub.vv         v10, v4, v10
+    vmax.vv         v8, v8, v25
+    vmax.vv         v10, v10, v25
+    vmin.vv         v8, v8, v26
+    vmin.vv         v10, v10, v26
+
+#if HIGH_BIT_DEPTH
+    vsse16.v        v8, (t0), a1
+    vsse16.v        v10, (t1), a1
+#else
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vnsrl.wi        v0, v8, 0
+    vnsrl.wi        v1, v10, 0
+    vsse8.v         v0, (t0), a1
+    vsse8.v         v1, (t1), a1
+#endif
+    ret
+endfunc
+
+//void pelFilterChroma_H_c(pixel *src, intptr_t srcStep, intptr_t offset, int32_t tc, int32_t maskP, int32_t maskQ)
+function PFX(pelFilterChroma_H_rvv)
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+#endif
+    slli            t3, a2, 1
+    sub             t0, a0, t3
+    sub             t1, a0, a2
+    add             t2, a0, a2
+#if HIGH_BIT_DEPTH
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vle16.v         v2, (t0)
+    vle16.v         v3, (t1)
+    vle16.v         v4, (a0)
+    vle16.v         v5, (t2)
+#else
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vle8.v          v6, (t0)
+    vle8.v          v7, (t1)
+    vle8.v          v8, (a0)
+    vle8.v          v9, (t2)
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vzext.vf2       v2, v6
+    vzext.vf2       v3, v7
+    vzext.vf2       v4, v8
+    vzext.vf2       v5, v9
+#endif
+    li              a6, (1 << BIT_DEPTH) - 1
+    neg             t3, a3
+    vmv.v.i         v27, 4
+    vmv.v.i         v25, 0
+    vmv.v.x         v28, a3
+    vmv.v.x         v29, t3
+    vmv.v.x         v30, a4
+    vmv.v.x         v31, a5
+    vmv.v.x         v26, a6
+    vsub.vv         v6, v4, v3
+    vsub.vv         v7, v2, v5
+    vmacc.vv        v7, v6, v27
+    vadd.vi         v7, v7, 4
+    vsra.vi         v7, v7, 3
+    vmax.vv         v7, v7, v29
+    vmin.vv         v7, v7, v28
+
+    vand.vv         v8, v7, v30
+    vand.vv         v10, v7, v31
+    vadd.vv         v8, v8, v3
+    vsub.vv         v10, v4, v10
+    vmax.vv         v8, v8, v25
+    vmax.vv         v10, v10, v25
+    vmin.vv         v8, v8, v26
+    vmin.vv         v10, v10, v26
+
+#if HIGH_BIT_DEPTH
+    vse16.v         v8, (t1)
+    vse16.v         v10, (a0)
+#else
+    vsetivli        zero, 4, e8, m1, ta, ma
+    vnsrl.wi        v0, v8, 0
+    vnsrl.wi        v1, v10, 0
+    vse8.v          v0, (t1)
+    vse8.v          v1, (a0)
+#endif
+    ret
+endfunc

x265-4.2.tar/source/common/riscv64/mc-a.S Added

@@ -0,0 +1,112 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+.align 4
+
+.text
+
+#if BIT_DEPTH == 8
+// void pixelavg_pp(pixel *dst, intptr_t dstride, const pixel *src0, intptr_t sstride0,
+//                  const pixel *src1, intptr_t sstride1, int);
+.macro AVG_PP_LOOP_FUNC w, h, lmul
+function PFX(pixel_avg_pp_\w\()x\h\()_rvv)
+    csrwi        vxrm, 0
+    li           t0, \h
+    li           t1, \w
+1:
+    addi         t0, t0, -4
+    vsetvli      zero, t1, e8, m\()\lmul, ta, ma
+
+    vle8.v       v0, (a2)
+    vle8.v       v16, (a4)
+    add          a2, a2, a3
+    add          a4, a4, a5
+    vaaddu.vv    v0, v0, v16
+
+    vle8.v       v4, (a2)
+    vle8.v       v20, (a4)
+    add          a2, a2, a3
+    add          a4, a4, a5
+    vaaddu.vv    v4, v4, v20
+
+    vle8.v       v8, (a2)
+    vle8.v       v24, (a4)
+    add          a2, a2, a3
+    add          a4, a4, a5
+    vaaddu.vv    v8, v8, v24
+
+    vle8.v       v12, (a2)
+    vle8.v       v28, (a4)
+    add          a2, a2, a3
+    add          a4, a4, a5
+    vaaddu.vv    v12, v12, v28
+
+    vse8.v       v0, (a0)
+    add          a0, a0, a1
+    vse8.v       v4, (a0)
+    add          a0, a0, a1
+    vse8.v       v8, (a0)
+    add          a0, a0, a1
+    vse8.v       v12, (a0)
+    add          a0, a0, a1
+    bnez         t0, 1b
+    ret
+endfunc
+.endm
+
+AVG_PP_LOOP_FUNC  4,  4,  1
+AVG_PP_LOOP_FUNC  4,  8,  1
+AVG_PP_LOOP_FUNC  4,  16,  1
+
+AVG_PP_LOOP_FUNC  8,  4,  1
+AVG_PP_LOOP_FUNC  8,  8,  1
+AVG_PP_LOOP_FUNC  8,  16,  1
+AVG_PP_LOOP_FUNC  8,  32,  1
+
+AVG_PP_LOOP_FUNC  12,  16,  1
+
+AVG_PP_LOOP_FUNC  16,  4,  1
+AVG_PP_LOOP_FUNC  16,  8,  1
+AVG_PP_LOOP_FUNC  16,  12,  1
+AVG_PP_LOOP_FUNC  16,  16,  1
+AVG_PP_LOOP_FUNC  16,  32,  1
+AVG_PP_LOOP_FUNC  16,  64,  1
+
+AVG_PP_LOOP_FUNC  24,  32,  2
+
+AVG_PP_LOOP_FUNC  32,  8,  2
+AVG_PP_LOOP_FUNC  32,  16,  2
+AVG_PP_LOOP_FUNC  32,  24,  2
+AVG_PP_LOOP_FUNC  32,  32,  2
+AVG_PP_LOOP_FUNC  32,  64,  2
+
+AVG_PP_LOOP_FUNC  48,  64,  4
+
+AVG_PP_LOOP_FUNC  64,  16,  4
+AVG_PP_LOOP_FUNC  64,  32,  4
+AVG_PP_LOOP_FUNC  64,  48,  4
+AVG_PP_LOOP_FUNC  64,  64,  4
+
+#endif
\ No newline at end of file

x265-4.2.tar/source/common/riscv64/p2s.S Added

@@ -0,0 +1,138 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Yujiao He <he.yujiao@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+ #include "asm.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+
+.text
+
+#if HIGH_BIT_DEPTH
+# if BIT_DEPTH == 10
+#  define P2S_SHIFT 4
+# elif BIT_DEPTH == 12
+#  define P2S_SHIFT 2
+# endif
+#else // if !HIGH_BIT_DEPTH
+# define P2S_SHIFT 6
+#endif // HIGH_BIT_DEPTH
+
+.macro p2s_x1 lmul1, lmul2
+#if HIGH_BIT_DEPTH
+    vle16.v         v8, (a0)
+    vsll.vi         v16, v8, P2S_SHIFT
+#else
+    vsetvli         zero, t0, e8, \lmul2, ta, ma
+    vle8.v          v8, (a0)
+    vsetvli         zero, t0, e16, \lmul1, ta, ma
+    vzext.vf2       v16, v8
+    vsll.vi         v16, v16, P2S_SHIFT
+#endif
+    vadd.vv         v16, v16, v0
+    vse16.v         v16, (a2)
+.endm
+
+.macro P2S_FUNC w, h, lmul1, lmul2
+function PFX(filterPixelToShort_\w\()x\h\()_rvv)
+    li              t0, \w
+    li              t2, 0xe000
+    vsetvli         zero, t0, e16, \lmul1, ta, ma
+    vmv.v.x         v0, t2
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+#endif
+    slli            a3, a3, 1
+.if \h <= 16 /* height <= 16, fully unroll*/
+    p2s_x1          \lmul1, \lmul2
+.rept \h -1
+    add             a0, a0, a1
+    add             a2, a2, a3
+    p2s_x1          \lmul1, \lmul2
+.endr
+.else  /* height > 16, 4 rows unroll*/
+    li              t1, \h
+1:
+.rept 4
+    p2s_x1          \lmul1, \lmul2
+    add             a0, a0, a1
+    add             a2, a2, a3
+.endr
+    addi            t1, t1, -4
+    bnez            t1, 1b
+.endif
+    ret
+endfunc
+.endm
+
+P2S_FUNC 4, 2, m1, m1
+P2S_FUNC 4, 4, m1, m1
+P2S_FUNC 4, 8, m1, m1
+P2S_FUNC 4, 16, m1, m1
+P2S_FUNC 4, 32, m1, m1
+
+P2S_FUNC 6, 8, m1, m1
+P2S_FUNC 6, 16, m1, m1
+
+P2S_FUNC 8, 2, m1, m1
+P2S_FUNC 8, 4, m1, m1
+P2S_FUNC 8, 8, m1, m1
+P2S_FUNC 8, 6, m1, m1
+P2S_FUNC 8, 12, m1, m1
+P2S_FUNC 8, 16, m1, m1
+P2S_FUNC 8, 32, m1, m1
+P2S_FUNC 8, 64, m1, m1
+
+P2S_FUNC 12, 16, m2, m1
+P2S_FUNC 12, 32, m2, m1
+
+P2S_FUNC 16, 4, m2, m1
+P2S_FUNC 16, 8, m2, m1
+P2S_FUNC 16, 12, m2, m1
+P2S_FUNC 16, 16, m2, m1
+P2S_FUNC 16, 24, m2, m1
+P2S_FUNC 16, 32, m2, m1
+P2S_FUNC 16, 64, m2, m1
+
+P2S_FUNC 24, 32, m4, m2
+P2S_FUNC 24, 64, m4, m2
+
+P2S_FUNC 32, 8, m4, m2
+P2S_FUNC 32, 12, m4, m2
+P2S_FUNC 32, 16, m4, m2
+P2S_FUNC 32, 24, m4, m2
+P2S_FUNC 32, 32, m4, m2
+P2S_FUNC 32, 48, m4, m2
+P2S_FUNC 32, 64, m4, m2
+
+P2S_FUNC 48, 64, m8, m4
+
+P2S_FUNC 64, 16, m8, m4
+P2S_FUNC 64, 32, m8, m4
+P2S_FUNC 64, 48, m8, m4
+P2S_FUNC 64, 64, m8, m4

x265-4.2.tar/source/common/riscv64/pixel-prim.cpp Added

@@ -0,0 +1,1497 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *          foolgry <wang.zhiyong11@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "common.h"
+#include "slicetype.h" // LOWRES_COST_MASK
+#include "primitives.h"
+#include "x265.h"
+#include "riscv64_utils.h"
+
+#include <riscv_vector.h>
+#include <stdint.h>
+
+#define SUMSUB_AB(sum, diff, a, b, vl)                     \
+    do {                                                   \
+        vuint16m1_t _a = (a);                              \
+        vuint16m1_t _b = (b);                              \
+        (sum) = __riscv_vadd_vv_u16m1(_a, _b, (vl));       \
+        (diff) = __riscv_vsub_vv_u16m1(_a, _b, (vl));      \
+    } while (0)
+
+namespace {
+
+using namespace X265_NS;
+
+#if HIGH_BIT_DEPTH
+// todo
+
+#else // !HIGH_BIT_DEPTH
+
+typedef struct {
+    vuint8mf2_t *m0;
+    vuint8mf2_t *m1;
+    vuint8mf2_t *m2;
+    vuint8mf2_t *m3;
+} vectors_u8_mf2_t;
+
+typedef struct {
+    vuint8mf2_t *m0;
+    vuint8mf2_t *m1;
+    vuint8mf2_t *m2;
+    vuint8mf2_t *m3;
+    vuint8mf2_t *m4;
+    vuint8mf2_t *m5;
+    vuint8mf2_t *m6;
+    vuint8mf2_t *m7;
+} vectors_u8_mf2_8_t;
+
+typedef struct {
+    vuint8m1_t *m0;
+    vuint8m1_t *m1;
+    vuint8m1_t *m2;
+    vuint8m1_t *m3;
+} vectors_u8_m1_t;
+
+typedef struct {
+    vuint16m1_t *m0;
+    vuint16m1_t *m1;
+    vuint16m1_t *m2;
+    vuint16m1_t *m3;
+} vectors_u16_m1_t;
+
+typedef struct {
+    vuint16m1_t *m0;
+    vuint16m1_t *m1;
+    vuint16m1_t *m2;
+    vuint16m1_t *m3;
+    vuint16m1_t *m4;
+    vuint16m1_t *m5;
+    vuint16m1_t *m6;
+    vuint16m1_t *m7;
+} vectors_u16_m1_8_t;
+
+// 4 times vle8 performs better than one time vlsseg4e8.
+// 1. array elements cannot have RVV type, such as 'vuint8mf2_t';
+// 2. The member variables in the structure cannot be RVV type, so use the structure pointer version.
+static void inline vload_u8x8x4_mf2(const uint8_t **pix, const intptr_t stride_pix,
+                               vectors_u8_mf2_t *d, size_t vl)
+{
+    *(d->m0) = __riscv_vle8_v_u8mf2(*pix, vl);
+    *(d->m1) = __riscv_vle8_v_u8mf2(*pix + stride_pix, vl);
+    *(d->m2) = __riscv_vle8_v_u8mf2(*pix + 2 * stride_pix, vl);
+    *(d->m3) = __riscv_vle8_v_u8mf2(*pix + 3 * stride_pix, vl);
+}
+
+static inline void load_diff_u8x8x8(const uint8_t *pix1, intptr_t stride_pix1,
+                                    const uint8_t *pix2, intptr_t stride_pix2, vectors_u16_m1_8_t *diff)
+{
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+
+    size_t vl = __riscv_vsetvl_e8mf2(8);
+    vuint8mf2_t r0, r1, r2, r3, t0, t1, t2, t3;
+    vectors_u8_mf2_t r = {&r0, &r1, &r2, &r3};
+    vectors_u8_mf2_t t = {&t0, &t1, &t2, &t3};
+
+    //row 0~3
+    vload_u8x8x4_mf2(&pix1_ptr, stride_pix1, &r, vl);
+    vload_u8x8x4_mf2(&pix2_ptr, stride_pix2, &t, vl);
+    *(diff->m0) = __riscv_vwsubu_vv_u16m1(*(r.m0), *(t.m0), vl);
+    *(diff->m1) = __riscv_vwsubu_vv_u16m1(*(r.m1), *(t.m1), vl);
+    *(diff->m2) = __riscv_vwsubu_vv_u16m1(*(r.m2), *(t.m2), vl);
+    *(diff->m3) = __riscv_vwsubu_vv_u16m1(*(r.m3), *(t.m3), vl);
+    //row4~7
+
+    pix1_ptr += 4 * stride_pix1;
+    pix2_ptr += 4 * stride_pix2;
+    vload_u8x8x4_mf2(&pix1_ptr, stride_pix1, &r, vl);
+    vload_u8x8x4_mf2(&pix2_ptr, stride_pix2, &t, vl);
+    *(diff->m4) = __riscv_vwsubu_vv_u16m1(*(r.m0), *(t.m0), vl);
+    *(diff->m5) = __riscv_vwsubu_vv_u16m1(*(r.m1), *(t.m1), vl);
+    *(diff->m6) = __riscv_vwsubu_vv_u16m1(*(r.m2), *(t.m2), vl);
+    *(diff->m7) = __riscv_vwsubu_vv_u16m1(*(r.m3), *(t.m3), vl);
+}
+
+static inline void vload_u8mf2x4(vectors_u8_mf2_t *d, const uint8_t **pix, intptr_t stride_pix, size_t vl) {
+    *(d->m0) = __riscv_vle8_v_u8mf2(*pix, vl);
+    *(d->m1) = __riscv_vle8_v_u8mf2(*pix + stride_pix, vl);
+    *(d->m2) = __riscv_vle8_v_u8mf2(*pix + 2 * stride_pix, vl);
+    *(d->m3) = __riscv_vle8_v_u8mf2(*pix + 3 * stride_pix, vl);
+    *pix += 4 * stride_pix;
+}
+
+static inline void vload_u8m1x4(vectors_u8_m1_t *d, const uint8_t **pix, intptr_t stride_pix, size_t vl) {
+    *(d->m0) = __riscv_vle8_v_u8m1(*pix, vl);
+    *(d->m1) = __riscv_vle8_v_u8m1(*pix + stride_pix, vl);
+    *(d->m2) = __riscv_vle8_v_u8m1(*pix + 2 * stride_pix, vl);
+    *(d->m3) = __riscv_vle8_v_u8m1(*pix + 3 * stride_pix, vl);
+    *pix += 4 * stride_pix;
+}
+
+static inline void vslide_combine_u8(vuint8mf2_t *d0, vuint8mf2_t *d1, vectors_u8_mf2_t s) {
+    *d0 = __riscv_vslideup_vx_u8mf2(*(s.m0), *(s.m2), 4, 8);
+    *d1 = __riscv_vslideup_vx_u8mf2(*(s.m1), *(s.m3), 4, 8);
+}
+
+static inline void vslidedown_u8x4(vectors_u8_m1_t *d0, vectors_u8_m1_t *d1,
+                                   const vectors_u8_m1_t s0, const vectors_u8_m1_t s1, size_t vl) {
+    const size_t offset = 8;
+    *(d0->m0) = __riscv_vslidedown_vx_u8m1(*(s0.m0), offset, vl);
+    *(d0->m1) = __riscv_vslidedown_vx_u8m1(*(s0.m1), offset, vl);
+    *(d0->m2) = __riscv_vslidedown_vx_u8m1(*(s0.m2), offset, vl);
+    *(d0->m3) = __riscv_vslidedown_vx_u8m1(*(s0.m3), offset, vl);
+
+    *(d1->m0) = __riscv_vslidedown_vx_u8m1(*(s1.m0), offset, vl);
+    *(d1->m1) = __riscv_vslidedown_vx_u8m1(*(s1.m1), offset, vl);
+    *(d1->m2) = __riscv_vslidedown_vx_u8m1(*(s1.m2), offset, vl);
+    *(d1->m3) = __riscv_vslidedown_vx_u8m1(*(s1.m3), offset, vl);
+}
+
+static inline void vwsubu_u8x4(vectors_u16_m1_t *diff, const vectors_u8_mf2_t s0, const vectors_u8_mf2_t s1, size_t vl) {
+    *(diff->m0) = __riscv_vwsubu_vv_u16m1(*(s0.m0), *(s1.m0), vl);
+    *(diff->m1) = __riscv_vwsubu_vv_u16m1(*(s0.m1), *(s1.m1), vl);
+    *(diff->m2) = __riscv_vwsubu_vv_u16m1(*(s0.m2), *(s1.m2), vl);
+    *(diff->m3) = __riscv_vwsubu_vv_u16m1(*(s0.m3), *(s1.m3), vl);
+}
+
+static inline void vget_first8_u8m1(vectors_u8_mf2_t *d, const vectors_u8_m1_t s) {
+    *(d->m0) = __riscv_vlmul_trunc_v_u8m1_u8mf2(*(s.m0));
+    *(d->m1) = __riscv_vlmul_trunc_v_u8m1_u8mf2(*(s.m1));
+    *(d->m2) = __riscv_vlmul_trunc_v_u8m1_u8mf2(*(s.m2));
+    *(d->m3) = __riscv_vlmul_trunc_v_u8m1_u8mf2(*(s.m3));
+}
+
+static inline vint16m1_t vabs_u16(vuint16m1_t s0, size_t vl) {
+    vint16m1_t tmp = __riscv_vreinterpret_v_u16m1_i16m1(s0);
+    vint16m1_t t0 = __riscv_vrsub_vx_i16m1(tmp, 0, vl);  // t0 = -s0
+    return __riscv_vmax_vv_i16m1(tmp, t0, vl);           // d0 = max(s0, t0)
+}
+
+static inline vuint16m1_t vmax_abs_u16(vuint16m1_t s0, vuint16m1_t s1, size_t vl) {
+    vuint16m1_t b0 = __riscv_vreinterpret_v_i16m1_u16m1(vabs_u16(s0, vl));
+    vuint16m1_t b1 = __riscv_vreinterpret_v_i16m1_u16m1(vabs_u16(s1, vl));
+    return __riscv_vmaxu_vv_u16m1(b0, b1, vl);
+}
+
+static inline int vredsum_u16(const vuint16m1_t src, size_t vl) {
+    vuint16m1_t v_sum = __riscv_vmv_v_x_u16m1(0, vl);
+    v_sum = __riscv_vredsum_vs_u16m1_u16m1(src, v_sum, vl);
+    return __riscv_vmv_x_s_u16m1_u16(v_sum);
+}
+
+static inline int vredsum_u32(const vuint32m2_t src, size_t vl) {
+    vuint32m1_t v_sum = __riscv_vmv_v_x_u32m1(0, 4);
+    v_sum = __riscv_vredsum_vs_u32m2_u32m1(src, v_sum, vl);
+    return __riscv_vmv_x_s_u32m1_u32(v_sum);
+}
+
+static inline void vtrn_8h(vuint16m1_t *d0, vuint16m1_t *d1, vuint16m1_t s0, vuint16m1_t s1) {
+    size_t vl = __riscv_vsetvl_e32m1(4);
+    const size_t shift = 16;
+
+    vuint32m1_t s0_32 = __riscv_vreinterpret_v_u16m1_u32m1(s0);
+    vuint32m1_t s1_32 = __riscv_vreinterpret_v_u16m1_u32m1(s1);
+
+    vuint32m1_t t2 = __riscv_vsll_vx_u32m1(s0_32, shift, vl);
+    vuint32m1_t t0 = __riscv_vsrl_vx_u32m1(s0_32, shift, vl);
+
+    vuint32m1_t d0_32 = __riscv_vsll_vx_u32m1(s1_32, shift, vl);
+    vuint32m1_t t1 = __riscv_vsrl_vx_u32m1(s1_32, shift, vl);
+
+    vuint32m1_t d1_32 = __riscv_vsll_vx_u32m1(t1, shift, vl);
+    t2 = __riscv_vsrl_vx_u32m1(t2, shift, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    *d0 = __riscv_vreinterpret_v_u32m1_u16m1(__riscv_vor_vv_u32m1(d0_32, t2, vl / 2));
+    *d1 = __riscv_vreinterpret_v_u32m1_u16m1(__riscv_vor_vv_u32m1(d1_32, t0, vl / 2));
+}
+
+static inline void vtrn_4s(vuint16m1_t *d0, vuint16m1_t *d1, vuint16m1_t s0, vuint16m1_t s1) {
+    size_t vl = __riscv_vsetvl_e64m1(2);
+    const size_t shift = 32;
+
+    vuint64m1_t s0_64 = __riscv_vreinterpret_v_u16m1_u64m1(s0);
+    vuint64m1_t s1_64 = __riscv_vreinterpret_v_u16m1_u64m1(s1);
+
+    vuint64m1_t t2 = __riscv_vsll_vx_u64m1(s0_64, shift, vl);
+    vuint64m1_t t0 =  __riscv_vsrl_vx_u64m1(s0_64, shift, vl);
+
+    vuint64m1_t d0_64 = __riscv_vsll_vx_u64m1(s1_64, shift, vl);
+    vuint64m1_t t1 =  __riscv_vsrl_vx_u64m1(s1_64, shift, vl);
+
+    vuint64m1_t d1_64 = __riscv_vsll_vx_u64m1(t1, shift, vl);
+    t2 = __riscv_vsrl_vx_u64m1(t2, shift, vl);
+
+    vl = __riscv_vsetvl_e32m1(4);
+    *d0 = __riscv_vreinterpret_v_u64m1_u16m1(__riscv_vor_vv_u64m1(d0_64, t2, vl / 2));
+    *d1 = __riscv_vreinterpret_v_u64m1_u16m1(__riscv_vor_vv_u64m1(d1_64, t0, vl / 2));
+}
+
+static inline void vtrn_16s(vuint16m1_t *d0, vuint16m1_t *d1, vuint16m1_t s0, vuint16m1_t s1) {
+    size_t vl = __riscv_vsetvl_e16m1(8);
+    const size_t offset = 4;
+    vuint16m1_t v1_slide = s1;
+
+    *d0 = __riscv_vslideup_vx_u16m1(s0, s1, offset, vl);
+
+    vl = __riscv_vsetvl_e16m1(4);
+    *d1 = __riscv_vslidedown_vx_u16m1_tu(v1_slide, s0, offset, vl);
+}
+
+// 4 way hadamard vertical pass.
+static inline void hadamard_vx4(vectors_u16_m1_t *out, const vectors_u16_m1_t *in, size_t vl) {
+    vuint16m1_t s0, s1, d0, d1;
+
+    SUMSUB_AB(s0, d0, *(in->m0), *(in->m1), vl);
+    SUMSUB_AB(s1, d1, *(in->m2), *(in->m3), vl);
+
+    SUMSUB_AB(*(out->m0), *(out->m2), s0, s1, vl);
+    SUMSUB_AB(*(out->m1), *(out->m3), d0, d1, vl);
+}
+
+// 8 way hadamard vertical pass.
+static inline void hadamard_vx8(vectors_u16_m1_8_t *out, const vectors_u16_m1_8_t *in, size_t vl)
+{
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7;
+    vectors_u16_m1_t t0={&tmp0, &tmp1, &tmp2, &tmp3};
+    vectors_u16_m1_t t1={&tmp4, &tmp5, &tmp6, &tmp7};
+    vectors_u16_m1_t in0={in->m0, in->m1, in->m2, in->m3};
+    vectors_u16_m1_t in1={in->m4, in->m5, in->m6, in->m7};
+
+    hadamard_vx4(&t0, &in0 ,vl);
+    hadamard_vx4(&t1, &in1, vl);
+
+    SUMSUB_AB(*(out->m0), *(out->m4), *(t0.m0), *(t1.m0), vl);
+    SUMSUB_AB(*(out->m1), *(out->m5), *(t0.m1), *(t1.m1), vl);
+    SUMSUB_AB(*(out->m2), *(out->m6), *(t0.m2), *(t1.m2), vl);
+    SUMSUB_AB(*(out->m3), *(out->m7), *(t0.m3), *(t1.m3), vl);
+}
+
+// 4 way hadamard horizontal pass.
+static inline void hadamard_hx4(vectors_u16_m1_t *out, const vectors_u16_m1_t *in, size_t vl) {
+    vuint16m1_t s0, s1, t0, t1, t2, t3, d0, d1;
+
+    vtrn_8h(&t0, &t1, *(in->m0), *(in->m1));
+    vtrn_8h(&t2, &t3, *(in->m2), *(in->m3));
+
+    SUMSUB_AB(s0, d0, t0, t1, vl);
+    SUMSUB_AB(s1, d1, t2, t3, vl);
+
+    vtrn_4s(out->m0, out->m1, s0, s1);
+    vtrn_4s(out->m2, out->m3, d0, d1);
+}
+
+// 8 way hadamard horizontal pass.
+static inline void hadamard_hx8(vectors_u16_m1_t *out, const vectors_u16_m1_8_t *in, size_t vl)
+{
+    vuint16m1_t s0, s1, s2, s3, d0, d1, d2, d3;
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7;
+    vectors_u16_m1_t t0={&tmp0, &tmp1, &tmp2, &tmp3};
+    vectors_u16_m1_t t1={&tmp4, &tmp5, &tmp6, &tmp7};
+    vectors_u16_m1_t in0={in->m0, in->m1, in->m2, in->m3};
+    vectors_u16_m1_t in1={in->m4, in->m5, in->m6, in->m7};
+
+    hadamard_hx4(&t0, &in0 ,vl);
+    hadamard_hx4(&t1, &in1 ,vl);
+
+    SUMSUB_AB(s0, d0, *(t0.m0), *(t0.m1), vl);
+    SUMSUB_AB(s1, d1, *(t0.m2), *(t0.m3), vl);
+    SUMSUB_AB(s2, d2, *(t1.m0), *(t1.m1), vl);
+    SUMSUB_AB(s3, d3, *(t1.m2), *(t1.m3), vl);
+
+    vtrn_16s((t0.m0), (t0.m1), s0, s2);
+    vtrn_16s((t0.m2), (t0.m3), s1, s3);
+    vtrn_16s((t1.m0), (t1.m1), d0, d2);
+    vtrn_16s((t1.m2), (t1.m3), d1, d3);
+
+    *(out->m0) = vmax_abs_u16(*(t0.m0), *(t0.m1), vl);
+    *(out->m1) = vmax_abs_u16(*(t0.m2), *(t0.m3), vl);
+    *(out->m2) = vmax_abs_u16(*(t1.m0), *(t1.m1), vl);
+    *(out->m3) = vmax_abs_u16(*(t1.m2), *(t1.m3), vl);
+}
+
+// Calculate 2 4x4 hadamard transformation.
+static inline void hadamard_4x4x2(vuint16m1_t *out, vectors_u16_m1_t *s, size_t vl) {
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3;
+    vectors_u16_m1_t tmp = {&tmp0, &tmp1, &tmp2, &tmp3};
+
+    hadamard_vx4(&tmp, s, vl);
+    hadamard_hx4(s, &tmp, vl);
+
+    vuint16m1_t b0 = vmax_abs_u16(*(s->m0), *(s->m1), vl);
+    vuint16m1_t b1 = vmax_abs_u16(*(s->m2), *(s->m3), vl);
+    *out = __riscv_vadd_vv_u16m1(b0, b1, vl);
+}
+
+// Calculate 4 4x4 hadamard transformation.
+static inline void hadamard_4x4x4(vuint16m1_t *out0, vuint16m1_t *out1, vectors_u16_m1_t *s0,
+                                  vectors_u16_m1_t *s1, size_t vl) {
+    vuint16m1_t tmp0_0, tmp0_1, tmp0_2, tmp0_3, tmp1_0, tmp1_1, tmp1_2, tmp1_3;
+    vectors_u16_m1_t tmp0 = {&tmp0_0, &tmp0_1, &tmp0_2, &tmp0_3};
+    vectors_u16_m1_t tmp1 = {&tmp1_0, &tmp1_1, &tmp1_2, &tmp1_3};
+    hadamard_vx4(&tmp0, s0, vl);
+    hadamard_vx4(&tmp1, s1, vl);
+
+    hadamard_hx4(s0, &tmp0, vl);
+    hadamard_hx4(s1, &tmp1, vl);
+
+    vuint16m1_t b0 = vmax_abs_u16(*(s0->m0), *(s0->m1), vl);
+    vuint16m1_t b1 = vmax_abs_u16(*(s0->m2), *(s0->m3), vl);
+    vuint16m1_t b2 = vmax_abs_u16(*(s1->m0), *(s1->m1), vl);
+    vuint16m1_t b3 = vmax_abs_u16(*(s1->m2), *(s1->m3), vl);
+    *out0 = __riscv_vadd_vv_u16m1(b0, b1, vl);
+    *out1 = __riscv_vadd_vv_u16m1(b2, b3, vl);
+}
+
+// Calculate 8x8 hadamard transformation.
+static inline void hadamard_8x8(vuint16m1_t *out0, vuint16m1_t *out1, vectors_u16_m1_8_t *diff, size_t vl)
+{
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7;
+    vectors_u16_m1_8_t tmp={&tmp0, &tmp1, &tmp2, &tmp3, &tmp4, &tmp5, &tmp6, &tmp7};
+    vuint16m1_t sum0, sum1, sum2, sum3;
+    vectors_u16_m1_t sum={&sum0, &sum1, &sum2, &sum3};
+
+    hadamard_vx8(&tmp, diff, vl);
+    hadamard_hx8(&sum, &tmp, vl);
+
+    *out0 = __riscv_vadd_vv_u16m1(*(sum.m0), *(sum.m1), vl);
+    *out1 = __riscv_vadd_vv_u16m1(*(sum.m2), *(sum.m3), vl);
+}
+
+static inline int pixel_satd_4x4_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                     const uint8_t *pix2, intptr_t stride_pix2) {
+    // Load 4x4 blocks
+    size_t vl = __riscv_vsetvl_e8mf2(4);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint8mf2_t s0, s1, s2, s3, r0, r1, r2, r3;
+    vectors_u8_mf2_t s = {&s0, &s1, &s2, &s3};
+    vectors_u8_mf2_t r = {&r0, &r1, &r2, &r3};
+    vload_u8mf2x4(&s, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r, &pix2_ptr, stride_pix2, vl);
+
+    // Slide upper parts
+    vuint8mf2_t s_combined0, s_combined1, r_combined0, r_combined1;
+    vslide_combine_u8(&s_combined0, &s_combined1, s);
+    vslide_combine_u8(&r_combined0, &r_combined1, r);
+
+    // Convert to 16-bit and compute differences (sign extension, not zero extension)
+    vl = __riscv_vsetvl_e8mf2(8);
+    vuint16m1_t diff0 = __riscv_vwsubu_vv_u16m1(s_combined0, r_combined0, vl);
+    vuint16m1_t diff1 = __riscv_vwsubu_vv_u16m1(s_combined1, r_combined1, vl);
+
+    vuint16m1_t v0, v1, v2, v3;
+    vl = __riscv_vsetvl_e16m1(8);
+    SUMSUB_AB(v2, v3, diff0, diff1, vl);
+
+    // Slide and combine
+    vuint16m1_t v0_slide = __riscv_vslideup_vx_u16m1(v2, v3, 4, vl);
+    vuint16m1_t v1_slide = v3;
+    vl = __riscv_vsetvl_e16m1(4);
+    //v1_slide = __riscv_vslidedown_vx_u16m1(v2, 4, vl);
+    // Protect the tail of the destination register
+    v1_slide = __riscv_vslidedown_vx_u16m1_tu(v1_slide, v2, 4, vl);
+    vl = __riscv_vsetvl_e16m1(8);
+    SUMSUB_AB(v2, v3, v0_slide, v1_slide, vl);
+
+    vtrn_8h(&v0, &v1, v2, v3);
+    SUMSUB_AB(v2, v3, v0, v1, vl);
+
+    vtrn_4s(&v0, &v1, v2, v3);
+
+    // Absolute values and reduction
+    vl = __riscv_vsetvl_e16m1(8);
+    vuint16m1_t vmax = vmax_abs_u16(v0, v1, vl);
+    int sum = vredsum_u16(vmax, vl);
+    return sum;
+}
+
+static inline int pixel_satd_4x8_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                     const uint8_t *pix2, intptr_t stride_pix2) {
+    // Load 4x8 blocks
+    size_t vl = __riscv_vsetvl_e8mf2(4);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+
+    // row: 0-3
+    vuint8mf2_t s0_0, s0_1, s0_2, s0_3;
+    vuint8mf2_t r0_0, r0_1, r0_2, r0_3;
+    vectors_u8_mf2_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_mf2_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vload_u8mf2x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r0, &pix2_ptr, stride_pix2, vl);
+
+    // row: 4-7
+    vuint8mf2_t s1_0, s1_1, s1_2, s1_3;
+    vuint8mf2_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_mf2_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_mf2_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vload_u8mf2x4(&s1, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r1, &pix2_ptr, stride_pix2, vl);
+
+    // Slide upper parts
+    vuint8mf2_t s_combined0, s_combined1, s_combined2, s_combined3;
+    vuint8mf2_t r_combined0, r_combined1, r_combined2, r_combined3;
+    vectors_u8_mf2_t tmp_s0 = {s0.m0, s0.m1, s1.m0, s1.m1};
+    vectors_u8_mf2_t tmp_s1 = {s0.m2, s0.m3, s1.m2, s1.m3};
+    vectors_u8_mf2_t tmp_r0 = {r0.m0, r0.m1, r1.m0, r1.m1};
+    vectors_u8_mf2_t tmp_r1 = {r0.m2, r0.m3, r1.m2, r1.m3};
+    vslide_combine_u8(&s_combined0, &s_combined1, tmp_s0);
+    vslide_combine_u8(&s_combined2, &s_combined3, tmp_s1);
+    vslide_combine_u8(&r_combined0, &r_combined1, tmp_r0);
+    vslide_combine_u8(&r_combined2, &r_combined3, tmp_r1);
+
+    // Convert to 16-bit and compute differences
+    vectors_u8_mf2_t tmp_s_combined = {&s_combined0, &s_combined1, &s_combined2, &s_combined3};
+    vectors_u8_mf2_t tmp_r_combined = {&r_combined0, &r_combined1, &r_combined2, &r_combined3};
+    vuint16m1_t diff0, diff1, diff2, diff3, out;
+    vectors_u16_m1_t diff = {&diff0, &diff1, &diff2, &diff3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff, tmp_s_combined, tmp_r_combined, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x2(&out, &diff, vl);
+    int sum = vredsum_u16(out, vl);
+
+    return sum;
+}
+
+static inline int pixel_satd_8x4_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                     const uint8_t *pix2, intptr_t stride_pix2) {
+    // Load 8x4 blocks, convert to 16-bit and compute differences
+    size_t vl = __riscv_vsetvl_e8mf2(8);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint8mf2_t s0, s1, s2, s3, r0, r1, r2, r3;
+    vectors_u8_mf2_t s = {&s0, &s1, &s2, &s3};
+    vectors_u8_mf2_t r = {&r0, &r1, &r2, &r3};
+    vload_u8mf2x4(&s, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r, &pix2_ptr, stride_pix2, vl);
+    vuint16m1_t diff0, diff1, diff2, diff3, out;
+    vectors_u16_m1_t diff = {&diff0, &diff1, &diff2, &diff3};
+    vwsubu_u8x4(&diff, s, r, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x2(&out, &diff, vl);
+    int sum = vredsum_u16(out, vl);
+
+    return sum;
+}
+
+static inline int pixel_satd_8x8_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                     const uint8_t *pix2, intptr_t stride_pix2) {
+    // Load 8x8 blocks; convert to 16-bit and compute differences
+    size_t vl = __riscv_vsetvl_e8mf2(8);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    // row: 0-3
+    vuint8mf2_t s0_0, s0_1, s0_2, s0_3;
+    vuint8mf2_t r0_0, r0_1, r0_2, r0_3;
+    vuint16m1_t diff0_0, diff0_1, diff0_2, diff0_3;
+    vectors_u8_mf2_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_mf2_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vectors_u16_m1_t diff0 = {&diff0_0, &diff0_1, &diff0_2, &diff0_3};
+    vload_u8mf2x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r0, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff0, s0, r0, vl);
+
+    // row: 4-7
+    vuint8mf2_t s1_0, s1_1, s1_2, s1_3;
+    vuint8mf2_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_mf2_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_mf2_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vuint16m1_t diff1_0, diff1_1, diff1_2, diff1_3;
+    vectors_u16_m1_t diff1 = {&diff1_0, &diff1_1, &diff1_2, &diff1_3};
+    vload_u8mf2x4(&s1, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r1, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff1, s1, r1, vl);
+
+    vuint16m1_t out0, out1;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff0, &diff1, vl);
+
+    // Absolute values and reduction
+    out0 = __riscv_vadd_vv_u16m1(out0, out1, vl);
+    int sum = vredsum_u16(out0, vl);
+
+    return sum;
+}
+
+static inline int pixel_satd_8x16_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2) {
+    // Load 8x16 blocks
+    size_t vl = __riscv_vsetvl_e8mf2(8);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    // row: 0-3
+    vuint8mf2_t s0_0, s0_1, s0_2, s0_3;
+    vuint8mf2_t r0_0, r0_1, r0_2, r0_3;
+    vuint16m1_t diff0_0, diff0_1, diff0_2, diff0_3;
+    vectors_u8_mf2_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_mf2_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vectors_u16_m1_t diff0 = {&diff0_0, &diff0_1, &diff0_2, &diff0_3};
+    vload_u8mf2x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r0, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff0, s0, r0, vl);
+    // row: 4-7
+    vuint8mf2_t s1_0, s1_1, s1_2, s1_3;
+    vuint8mf2_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_mf2_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_mf2_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vuint16m1_t diff1_0, diff1_1, diff1_2, diff1_3;
+    vectors_u16_m1_t diff1 = {&diff1_0, &diff1_1, &diff1_2, &diff1_3};
+    vload_u8mf2x4(&s1, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r1, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff1, s1, r1, vl);
+
+    vuint16m1_t out0, out1;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff0, &diff1, vl);
+
+    // row: 8-11
+    vl = __riscv_vsetvl_e8mf2(8);
+    vuint8mf2_t s2_0, s2_1, s2_2, s2_3;
+    vuint8mf2_t r2_0, r2_1, r2_2, r2_3;
+    vuint16m1_t diff2_0, diff2_1, diff2_2, diff2_3;
+    vectors_u8_mf2_t s2 = {&s2_0, &s2_1, &s2_2, &s2_3};
+    vectors_u8_mf2_t r2 = {&r2_0, &r2_1, &r2_2, &r2_3};
+    vectors_u16_m1_t diff2 = {&diff2_0, &diff2_1, &diff2_2, &diff2_3};
+    vload_u8mf2x4(&s2, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r2, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff2, s2, r2, vl);
+    // row: 12-15
+    vuint8mf2_t s3_0, s3_1, s3_2, s3_3;
+    vuint8mf2_t r3_0, r3_1, r3_2, r3_3;
+    vectors_u8_mf2_t s3 = {&s3_0, &s3_1, &s3_2, &s3_3};
+    vectors_u8_mf2_t r3 = {&r3_0, &r3_1, &r3_2, &r3_3};
+    vuint16m1_t diff3_0, diff3_1, diff3_2, diff3_3;
+    vectors_u16_m1_t diff3 = {&diff3_0, &diff3_1, &diff3_2, &diff3_3};
+    vload_u8mf2x4(&s3, &pix1_ptr, stride_pix1, vl);
+    vload_u8mf2x4(&r3, &pix2_ptr, stride_pix2, vl);
+    vwsubu_u8x4(&diff3, s3, r3, vl);
+
+    vuint16m1_t out2, out3;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out2, &out3, &diff2, &diff3, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    vuint16m1_t sum0 = __riscv_vadd_vv_u16m1(out0, out1, vl);
+    vuint16m1_t sum1 = __riscv_vadd_vv_u16m1(out2, out3, vl);
+    vuint16m1_t sum2 = __riscv_vadd_vv_u16m1(sum0, sum1, vl);
+    int sum = vredsum_u16(sum2, vl);
+
+    return sum;
+}
+
+// To be optimized
+static inline int pixel_satd_16x4_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2) {
+/*
+|       8 element     |    |       8 element      |
+|       diff00      |    |       diff10       |
+|       diff01      |    |       diff11       |
+|       diff02      |    |       diff12       |
+|       diff03      |    |       diff13       |
+*/
+    size_t vl = __riscv_vsetvl_e8m1(16);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint8m1_t s0_0, s0_1, s0_2, s0_3;
+    vuint8m1_t r0_0, r0_1, r0_2, r0_3;
+    vectors_u8_m1_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_m1_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vload_u8m1x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r0, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s0_h_0, s0_h_1, s0_h_2, s0_h_3;
+    vuint8mf2_t r0_h_0, r0_h_1, r0_h_2, r0_h_3;
+    vectors_u8_mf2_t s0_half = {&s0_h_0, &s0_h_1, &s0_h_2, &s0_h_3};
+    vectors_u8_mf2_t r0_half = {&r0_h_0, &r0_h_1, &r0_h_2, &r0_h_3};
+    vget_first8_u8m1(&s0_half, s0);
+    vget_first8_u8m1(&r0_half, r0);
+    vuint16m1_t diff0_0, diff0_1, diff0_2, diff0_3;
+    vectors_u16_m1_t diff0 = {&diff0_0, &diff0_1, &diff0_2, &diff0_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff0, s0_half, r0_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s1_0, s1_1, s1_2, s1_3;
+    vuint8m1_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_m1_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_m1_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vslidedown_u8x4(&s1, &r1, s0, r0, vl);
+    vuint8mf2_t s1_h_0, s1_h_1, s1_h_2, s1_h_3;
+    vuint8mf2_t r1_h_0, r1_h_1, r1_h_2, r1_h_3;
+    vectors_u8_mf2_t s1_half = {&s1_h_0, &s1_h_1, &s1_h_2, &s1_h_3};
+    vectors_u8_mf2_t r1_half = {&r1_h_0, &r1_h_1, &r1_h_2, &r1_h_3};
+    vget_first8_u8m1(&s1_half, s1);
+    vget_first8_u8m1(&r1_half, r1);
+    vuint16m1_t diff1_0, diff1_1, diff1_2, diff1_3;
+    vectors_u16_m1_t diff1 = {&diff1_0, &diff1_1, &diff1_2, &diff1_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff1, s1_half, r1_half, vl);
+
+    vuint16m1_t out0, out1;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x2(&out0, &diff0, vl);
+    hadamard_4x4x2(&out1, &diff1, vl);
+
+    out0 = __riscv_vadd_vv_u16m1(out0, out1, vl);
+    int sum = vredsum_u16(out0, vl);
+
+    return sum;
+}
+
+static inline int pixel_satd_16x8_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                       const uint8_t *pix2, intptr_t stride_pix2) {
+/*
+|       8 element     |    |       8 element      |
+|       diff00      |    |       diff10       |
+|       diff01      |    |       diff11       |
+|       diff02      |    |       diff12       |
+|       diff03      |    |       diff13       |
+-------------------------------------------------------
+|       diff20      |    |       diff30       |
+|       diff21      |    |       diff31       |
+|       diff22      |    |       diff32       |
+|       diff23      |    |       diff33       |
+*/
+    // row 0-3
+    size_t vl = __riscv_vsetvl_e8m1(16);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint8m1_t s0_0, s0_1, s0_2, s0_3;
+    vuint8m1_t r0_0, r0_1, r0_2, r0_3;
+    vectors_u8_m1_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_m1_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vload_u8m1x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r0, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s0_h_0, s0_h_1, s0_h_2, s0_h_3;
+    vuint8mf2_t r0_h_0, r0_h_1, r0_h_2, r0_h_3;
+    vectors_u8_mf2_t s0_half = {&s0_h_0, &s0_h_1, &s0_h_2, &s0_h_3};
+    vectors_u8_mf2_t r0_half = {&r0_h_0, &r0_h_1, &r0_h_2, &r0_h_3};
+    vget_first8_u8m1(&s0_half, s0);
+    vget_first8_u8m1(&r0_half, r0);
+    vuint16m1_t diff0_0, diff0_1, diff0_2, diff0_3;
+    vectors_u16_m1_t diff0 = {&diff0_0, &diff0_1, &diff0_2, &diff0_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff0, s0_half, r0_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s1_0, s1_1, s1_2, s1_3;
+    vuint8m1_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_m1_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_m1_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vslidedown_u8x4(&s1, &r1, s0, r0, vl);
+    vuint8mf2_t s1_h_0, s1_h_1, s1_h_2, s1_h_3;
+    vuint8mf2_t r1_h_0, r1_h_1, r1_h_2, r1_h_3;
+    vectors_u8_mf2_t s1_half = {&s1_h_0, &s1_h_1, &s1_h_2, &s1_h_3};
+    vectors_u8_mf2_t r1_half = {&r1_h_0, &r1_h_1, &r1_h_2, &r1_h_3};
+    vget_first8_u8m1(&s1_half, s1);
+    vget_first8_u8m1(&r1_half, r1);
+    vuint16m1_t diff1_0, diff1_1, diff1_2, diff1_3;
+    vectors_u16_m1_t diff1 = {&diff1_0, &diff1_1, &diff1_2, &diff1_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff1, s1_half, r1_half, vl);
+
+    vuint16m1_t out0, out1;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff0, &diff1, vl);
+
+    // row 4-7
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s2_0, s2_1, s2_2, s2_3;
+    vuint8m1_t r2_0, r2_1, r2_2, r2_3;
+    vectors_u8_m1_t s2 = {&s2_0, &s2_1, &s2_2, &s2_3};
+    vectors_u8_m1_t r2 = {&r2_0, &r2_1, &r2_2, &r2_3};
+    vload_u8m1x4(&s2, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r2, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s2_h_0, s2_h_1, s2_h_2, s2_h_3;
+    vuint8mf2_t r2_h_0, r2_h_1, r2_h_2, r2_h_3;
+    vectors_u8_mf2_t s2_half = {&s2_h_0, &s2_h_1, &s2_h_2, &s2_h_3};
+    vectors_u8_mf2_t r2_half = {&r2_h_0, &r2_h_1, &r2_h_2, &r2_h_3};
+    vget_first8_u8m1(&s2_half, s2);
+    vget_first8_u8m1(&r2_half, r2);
+    vuint16m1_t diff2_0, diff2_1, diff2_2, diff2_3;
+    vectors_u16_m1_t diff2 = {&diff2_0, &diff2_1, &diff2_2, &diff2_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff2, s2_half, r2_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s3_0, s3_1, s3_2, s3_3;
+    vuint8m1_t r3_0, r3_1, r3_2, r3_3;
+    vectors_u8_m1_t s3 = {&s3_0, &s3_1, &s3_2, &s3_3};
+    vectors_u8_m1_t r3 = {&r3_0, &r3_1, &r3_2, &r3_3};
+    vslidedown_u8x4(&s3, &r3, s2, r2, vl);
+    vuint8mf2_t s3_h_0, s3_h_1, s3_h_2, s3_h_3;
+    vuint8mf2_t r3_h_0, r3_h_1, r3_h_2, r3_h_3;
+    vectors_u8_mf2_t s3_half = {&s3_h_0, &s3_h_1, &s3_h_2, &s3_h_3};
+    vectors_u8_mf2_t r3_half = {&r3_h_0, &r3_h_1, &r3_h_2, &r3_h_3};
+    vget_first8_u8m1(&s3_half, s3);
+    vget_first8_u8m1(&r3_half, r3);
+    vuint16m1_t diff3_0, diff3_1, diff3_2, diff3_3;
+    vectors_u16_m1_t diff3 = {&diff3_0, &diff3_1, &diff3_2, &diff3_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff3, s3_half, r3_half, vl);
+
+    vuint16m1_t out2, out3;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out2, &out3, &diff2, &diff3, vl);
+
+    vuint16m1_t sum0 = __riscv_vadd_vv_u16m1(out0, out1, vl);
+    vuint16m1_t sum1 = __riscv_vadd_vv_u16m1(out2, out3, vl);
+    vuint16m1_t sum2 = __riscv_vadd_vv_u16m1(sum0, sum1, vl);
+    int sum = vredsum_u16(sum2, vl);
+
+    return sum;
+}
+
+static inline int pixel_satd_16x16_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                       const uint8_t *pix2, intptr_t stride_pix2) {
+/*
+|       8 element     |    |       8 element      |
+|       diff00      |    |       diff10       |
+|       diff01      |    |       diff11       |
+|       diff02      |    |       diff12       |
+|       diff03      |    |       diff13       |
+                        ...
+*/
+    // Load 16x16 blocks four times
+    // 1:
+    size_t vl = __riscv_vsetvl_e8m1(16);
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint8m1_t s0_0, s0_1, s0_2, s0_3;
+    vuint8m1_t r0_0, r0_1, r0_2, r0_3;
+    vectors_u8_m1_t s0 = {&s0_0, &s0_1, &s0_2, &s0_3};
+    vectors_u8_m1_t r0 = {&r0_0, &r0_1, &r0_2, &r0_3};
+    vload_u8m1x4(&s0, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r0, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s0_h_0, s0_h_1, s0_h_2, s0_h_3;
+    vuint8mf2_t r0_h_0, r0_h_1, r0_h_2, r0_h_3;
+    vectors_u8_mf2_t s0_half = {&s0_h_0, &s0_h_1, &s0_h_2, &s0_h_3};
+    vectors_u8_mf2_t r0_half = {&r0_h_0, &r0_h_1, &r0_h_2, &r0_h_3};
+    vget_first8_u8m1(&s0_half, s0);
+    vget_first8_u8m1(&r0_half, r0);
+    vuint16m1_t diff0_0, diff0_1, diff0_2, diff0_3;
+    vectors_u16_m1_t diff0 = {&diff0_0, &diff0_1, &diff0_2, &diff0_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff0, s0_half, r0_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s1_0, s1_1, s1_2, s1_3;
+    vuint8m1_t r1_0, r1_1, r1_2, r1_3;
+    vectors_u8_m1_t s1 = {&s1_0, &s1_1, &s1_2, &s1_3};
+    vectors_u8_m1_t r1 = {&r1_0, &r1_1, &r1_2, &r1_3};
+    vslidedown_u8x4(&s1, &r1, s0, r0, vl);
+    vuint8mf2_t s1_h_0, s1_h_1, s1_h_2, s1_h_3;
+    vuint8mf2_t r1_h_0, r1_h_1, r1_h_2, r1_h_3;
+    vectors_u8_mf2_t s1_half = {&s1_h_0, &s1_h_1, &s1_h_2, &s1_h_3};
+    vectors_u8_mf2_t r1_half = {&r1_h_0, &r1_h_1, &r1_h_2, &r1_h_3};
+    vget_first8_u8m1(&s1_half, s1);
+    vget_first8_u8m1(&r1_half, r1);
+    vuint16m1_t diff1_0, diff1_1, diff1_2, diff1_3;
+    vectors_u16_m1_t diff1 = {&diff1_0, &diff1_1, &diff1_2, &diff1_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff1, s1_half, r1_half, vl);
+
+    vuint16m1_t out0, out1, sum0, sum1;
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff0, &diff1, vl);
+    sum0 = out0;
+    sum1 = out1;
+
+    // 2:
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s2_0, s2_1, s2_2, s2_3;
+    vuint8m1_t r2_0, r2_1, r2_2, r2_3;
+    vectors_u8_m1_t s2 = {&s2_0, &s2_1, &s2_2, &s2_3};
+    vectors_u8_m1_t r2 = {&r2_0, &r2_1, &r2_2, &r2_3};
+    vload_u8m1x4(&s2, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r2, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s2_h_0, s2_h_1, s2_h_2, s2_h_3;
+    vuint8mf2_t r2_h_0, r2_h_1, r2_h_2, r2_h_3;
+    vectors_u8_mf2_t s2_half = {&s2_h_0, &s2_h_1, &s2_h_2, &s2_h_3};
+    vectors_u8_mf2_t r2_half = {&r2_h_0, &r2_h_1, &r2_h_2, &r2_h_3};
+    vget_first8_u8m1(&s2_half, s2);
+    vget_first8_u8m1(&r2_half, r2);
+    vuint16m1_t diff2_0, diff2_1, diff2_2, diff2_3;
+    vectors_u16_m1_t diff2 = {&diff2_0, &diff2_1, &diff2_2, &diff2_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff2, s2_half, r2_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s3_0, s3_1, s3_2, s3_3;
+    vuint8m1_t r3_0, r3_1, r3_2, r3_3;
+    vectors_u8_m1_t s3 = {&s3_0, &s3_1, &s3_2, &s3_3};
+    vectors_u8_m1_t r3 = {&r3_0, &r3_1, &r3_2, &r3_3};
+    vslidedown_u8x4(&s3, &r3, s2, r2, vl);
+    vuint8mf2_t s3_h_0, s3_h_1, s3_h_2, s3_h_3;
+    vuint8mf2_t r3_h_0, r3_h_1, r3_h_2, r3_h_3;
+    vectors_u8_mf2_t s3_half = {&s3_h_0, &s3_h_1, &s3_h_2, &s3_h_3};
+    vectors_u8_mf2_t r3_half = {&r3_h_0, &r3_h_1, &r3_h_2, &r3_h_3};
+    vget_first8_u8m1(&s3_half, s3);
+    vget_first8_u8m1(&r3_half, r3);
+    vuint16m1_t diff3_0, diff3_1, diff3_2, diff3_3;
+    vectors_u16_m1_t diff3 = {&diff3_0, &diff3_1, &diff3_2, &diff3_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff3, s3_half, r3_half, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff2, &diff3, vl);
+    sum0 = __riscv_vadd_vv_u16m1(sum0, out0, vl);
+    sum1 = __riscv_vadd_vv_u16m1(sum1, out1, vl);
+
+    // 3:
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s4_0, s4_1, s4_2, s4_3;
+    vuint8m1_t r4_0, r4_1, r4_2, r4_3;
+    vectors_u8_m1_t s4 = {&s4_0, &s4_1, &s4_2, &s4_3};
+    vectors_u8_m1_t r4 = {&r4_0, &r4_1, &r4_2, &r4_3};
+    vload_u8m1x4(&s4, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r4, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s4_h_0, s4_h_1, s4_h_2, s4_h_3;
+    vuint8mf2_t r4_h_0, r4_h_1, r4_h_2, r4_h_3;
+    vectors_u8_mf2_t s4_half = {&s4_h_0, &s4_h_1, &s4_h_2, &s4_h_3};
+    vectors_u8_mf2_t r4_half = {&r4_h_0, &r4_h_1, &r4_h_2, &r4_h_3};
+    vget_first8_u8m1(&s4_half, s4);
+    vget_first8_u8m1(&r4_half, r4);
+    vuint16m1_t diff4_0, diff4_1, diff4_2, diff4_3;
+    vectors_u16_m1_t diff4 = {&diff4_0, &diff4_1, &diff4_2, &diff4_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff4, s4_half, r4_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s5_0, s5_1, s5_2, s5_3;
+    vuint8m1_t r5_0, r5_1, r5_2, r5_3;
+    vectors_u8_m1_t s5 = {&s5_0, &s5_1, &s5_2, &s5_3};
+    vectors_u8_m1_t r5 = {&r5_0, &r5_1, &r5_2, &r5_3};
+    vslidedown_u8x4(&s5, &r5, s4, r4, vl);
+    vuint8mf2_t s5_h_0, s5_h_1, s5_h_2, s5_h_3;
+    vuint8mf2_t r5_h_0, r5_h_1, r5_h_2, r5_h_3;
+    vectors_u8_mf2_t s5_half = {&s5_h_0, &s5_h_1, &s5_h_2, &s5_h_3};
+    vectors_u8_mf2_t r5_half = {&r5_h_0, &r5_h_1, &r5_h_2, &r5_h_3};
+    vget_first8_u8m1(&s5_half, s5);
+    vget_first8_u8m1(&r5_half, r5);
+    vuint16m1_t diff5_0, diff5_1, diff5_2, diff5_3;
+    vectors_u16_m1_t diff5 = {&diff5_0, &diff5_1, &diff5_2, &diff5_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff5, s5_half, r5_half, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff4, &diff5, vl);
+    sum0 = __riscv_vadd_vv_u16m1(sum0, out0, vl);
+    sum1 = __riscv_vadd_vv_u16m1(sum1, out1, vl);
+
+    // 4:
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s6_0, s6_1, s6_2, s6_3;
+    vuint8m1_t r6_0, r6_1, r6_2, r6_3;
+    vectors_u8_m1_t s6 = {&s6_0, &s6_1, &s6_2, &s6_3};
+    vectors_u8_m1_t r6 = {&r6_0, &r6_1, &r6_2, &r6_3};
+    vload_u8m1x4(&s6, &pix1_ptr, stride_pix1, vl);
+    vload_u8m1x4(&r6, &pix2_ptr, stride_pix2, vl);
+
+    // Low part
+    vuint8mf2_t s6_h_0, s6_h_1, s6_h_2, s6_h_3;
+    vuint8mf2_t r6_h_0, r6_h_1, r6_h_2, r6_h_3;
+    vectors_u8_mf2_t s6_half = {&s6_h_0, &s6_h_1, &s6_h_2, &s6_h_3};
+    vectors_u8_mf2_t r6_half = {&r6_h_0, &r6_h_1, &r6_h_2, &r6_h_3};
+    vget_first8_u8m1(&s6_half, s6);
+    vget_first8_u8m1(&r6_half, r6);
+    vuint16m1_t diff6_0, diff6_1, diff6_2, diff6_3;
+    vectors_u16_m1_t diff6 = {&diff6_0, &diff6_1, &diff6_2, &diff6_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff6, s6_half, r6_half, vl);
+
+    // High part
+    vl = __riscv_vsetvl_e8m1(16);
+    vuint8m1_t s7_0, s7_1, s7_2, s7_3;
+    vuint8m1_t r7_0, r7_1, r7_2, r7_3;
+    vectors_u8_m1_t s7 = {&s7_0, &s7_1, &s7_2, &s7_3};
+    vectors_u8_m1_t r7 = {&r7_0, &r7_1, &r7_2, &r7_3};
+    vslidedown_u8x4(&s7, &r7, s6, r6, vl);
+    vuint8mf2_t s7_h_0, s7_h_1, s7_h_2, s7_h_3;
+    vuint8mf2_t r7_h_0, r7_h_1, r7_h_2, r7_h_3;
+    vectors_u8_mf2_t s7_half = {&s7_h_0, &s7_h_1, &s7_h_2, &s7_h_3};
+    vectors_u8_mf2_t r7_half = {&r7_h_0, &r7_h_1, &r7_h_2, &r7_h_3};
+    vget_first8_u8m1(&s7_half, s7);
+    vget_first8_u8m1(&r7_half, r7);
+    vuint16m1_t diff7_0, diff7_1, diff7_2, diff7_3;
+    vectors_u16_m1_t diff7 = {&diff7_0, &diff7_1, &diff7_2, &diff7_3};
+    vl = __riscv_vsetvl_e8mf2(8);
+    vwsubu_u8x4(&diff7, s7_half, r7_half, vl);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    hadamard_4x4x4(&out0, &out1, &diff6, &diff7, vl);
+    sum0 = __riscv_vadd_vv_u16m1(sum0, out0, vl);
+    sum1 = __riscv_vadd_vv_u16m1(sum1, out1, vl);
+
+    // Reduced sum
+    vl = __riscv_vsetvl_e32m2(8);
+    vuint32m2_t sum_ext0 = __riscv_vzext_vf2_u32m2(sum0, vl);
+    vuint32m2_t sum_ext1 = __riscv_vzext_vf2_u32m2(sum1, vl);
+
+    vuint32m1_t v_sum = __riscv_vmv_v_x_u32m1(0, vl);
+    v_sum = __riscv_vredsum_vs_u32m2_u32m1(sum_ext0, v_sum, vl);
+    v_sum = __riscv_vredsum_vs_u32m2_u32m1(sum_ext1, v_sum, vl);
+    return __riscv_vmv_x_s_u32m1_u32(v_sum);
+}
+
+static inline int pixel_sa8d_8x8_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                      const uint8_t *pix2, intptr_t stride_pix2)
+{
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint16m1_t diff0, diff1, diff2, diff3, diff4, diff5, diff6, diff7;
+    vectors_u16_m1_8_t diff = {&diff0, &diff1, &diff2, &diff3, &diff4, &diff5, &diff6, &diff7};
+    vuint16m1_t res0, res1;
+    vuint16m1_t out;
+
+    load_diff_u8x8x8(pix1_ptr, stride_pix1, pix2_ptr, stride_pix2, &diff);
+
+    size_t vl = __riscv_vsetvl_e16m1(8);
+    hadamard_8x8(&res0, &res1, &diff, vl);
+    out = __riscv_vadd_vv_u16m1(res0, res1, vl);
+    int sum = vredsum_u16(out, vl);
+    return (sum + 1) >> 1;
+}
+
+static inline int pixel_sa8d_16x16_rvv(const uint8_t *pix1, intptr_t stride_pix1,
+                                        const uint8_t *pix2, intptr_t stride_pix2)
+{
+    const uint8_t *pix1_ptr = pix1;
+    const uint8_t *pix2_ptr = pix2;
+    vuint16m1_t diff0, diff1, diff2, diff3, diff4, diff5, diff6, diff7;
+    vectors_u16_m1_8_t diff = {&diff0, &diff1, &diff2, &diff3, &diff4, &diff5, &diff6, &diff7};
+    vuint16m1_t res0, res1;
+    vuint32m2_t sum, tmp;
+
+    load_diff_u8x8x8(pix1_ptr, stride_pix1, pix2, stride_pix2, &diff);
+    size_t vl = __riscv_vsetvl_e16m1(8);
+    hadamard_8x8(&res0, &res1, &diff, vl);
+    sum = __riscv_vwaddu_vv_u32m2(res0, res1, vl);
+
+    load_diff_u8x8x8(pix1_ptr + 8, stride_pix1, pix2_ptr + 8, stride_pix2, &diff);
+    hadamard_8x8(&res0, &res1, &diff, vl);
+    tmp = __riscv_vwaddu_vv_u32m2(res0, res1, vl);
+    sum = __riscv_vadd_vv_u32m2(tmp, sum, vl);
+
+    load_diff_u8x8x8(pix1_ptr + 8 * stride_pix1, stride_pix1, pix2_ptr + 8 * stride_pix2, stride_pix2, &diff);
+    hadamard_8x8(&res0, &res1, &diff, vl);
+    tmp = __riscv_vwaddu_vv_u32m2(res0, res1, vl);
+    sum = __riscv_vadd_vv_u32m2(tmp, sum, vl);
+
+    load_diff_u8x8x8(pix1_ptr + 8 * stride_pix1 + 8, stride_pix1, pix2_ptr + 8 * stride_pix2 + 8, stride_pix2, &diff);
+    hadamard_8x8(&res0, &res1, &diff, vl);
+    tmp = __riscv_vwaddu_vv_u32m2(res0, res1, vl);
+    sum = __riscv_vadd_vv_u32m2(tmp, sum, vl);
+
+    int sum_int = vredsum_u32(sum, vl);
+
+    return (sum_int + 1) >> 1;
+}
+
+#endif // HIGH_BIT_DEPTH
+
+// To be optimized
+template<int lx, int ly>
+void pixelavg_pp_rvv(pixel *dst, intptr_t dstride, const pixel *src0, intptr_t sstride0, const pixel *src1,
+                     intptr_t sstride1, int)
+{
+    // Use rnu rounding mode
+    const unsigned int vxrm = 0;
+    for (int y = 0; y < ly; y++)
+    {
+        int x = 0;
+        size_t vl;
+        for (; x < lx; x += vl) {
+#if HIGH_BIT_DEPTH
+            vl = __riscv_vsetvl_e16m1(lx - x);
+            vuint16m1_t in0 = __riscv_vle16_v_u16m1((const uint16_t*)(src0 + x), vl);
+            vuint16m1_t in1 = __riscv_vle16_v_u16m1((const uint16_t*)(src1 + x), vl);
+            // compute (a + b + 1) >> 1
+            vuint16m1_t avg = __riscv_vaaddu_vv_u16m1(in0, in1, vxrm, vl);
+            __riscv_vse16_v_u16m1((uint16_t*)(dst + x), avg, vl);
+#else
+            vl = __riscv_vsetvl_e8m1(lx - x);
+            vuint8m1_t in0 = __riscv_vle8_v_u8m1((const uint8_t*)(src0 + x), vl);
+            vuint8m1_t in1 = __riscv_vle8_v_u8m1((const uint8_t*)(src1 + x), vl);
+            // zero-extended
+            vuint16m2_t w_in0 = __riscv_vzext_vf2_u16m2(in0, vl);
+            vuint16m2_t w_in1 = __riscv_vzext_vf2_u16m2(in1, vl);
+            vuint16m2_t sum = __riscv_vadd_vv_u16m2(w_in0, w_in1, vl);
+            vuint8m1_t avg = __riscv_vnclipu_wx_u8m1(sum, 1, vxrm, vl);
+            __riscv_vse8_v_u8m1((uint8_t*)(dst + x), avg, vl);
+#endif
+        }
+        src0 += sstride0;
+        src1 += sstride1;
+        dst += dstride;
+    }
+}
+
+#if !(HIGH_BIT_DEPTH)
+template<int w, int h>
+int satd4_rvv(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+{
+    int satd = 0;
+
+    if (w == 4 && h == 4) {
+        satd = pixel_satd_4x4_rvv(pix1, stride_pix1, pix2, stride_pix2);
+    } else {
+        for (int row = 0; row < h; row += 8)
+            for (int col = 0; col < w; col += 4)
+                satd += pixel_satd_4x8_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                           pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+
+    return satd;
+}
+
+template<int w, int h>
+int satd8_rvv(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+{
+    int satd = 0;
+
+    if (w % 16 == 0 && h % 16 == 0)
+    {
+        for (int row = 0; row < h; row += 16)
+            for (int col = 0; col < w; col += 16)
+                satd += pixel_satd_16x16_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                              pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 8 == 0 && h % 16 == 0)
+    {
+        for (int row = 0; row < h; row += 16)
+            for (int col = 0; col < w; col += 8)
+                satd += pixel_satd_8x16_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 16 == 0 && h % 8 == 0)
+    {
+        for (int row = 0; row < h; row += 8)
+            for (int col = 0; col < w; col += 16)
+                satd += pixel_satd_16x8_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 16 == 0 && h % 4 == 0)
+    {
+        for (int row = 0; row < h; row += 4)
+            for (int col = 0; col < w; col += 16)
+                satd += pixel_satd_16x4_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                             pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else if (w % 8 == 0 && h % 8 == 0)
+    {
+        for (int row = 0; row < h; row += 8)
+            for (int col = 0; col < w; col += 8)
+                satd += pixel_satd_8x8_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                            pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+    else // w multiple of 8, h multiple of 4
+    {
+        for (int row = 0; row < h; row += 4)
+            for (int col = 0; col < w; col += 8)
+                satd += pixel_satd_8x4_rvv(pix1 + row * stride_pix1 + col, stride_pix1,
+                                            pix2 + row * stride_pix2 + col, stride_pix2);
+    }
+
+    return satd;
+}
+
+// Calculate sa8d in blocks of 8x8
+template<int w, int h>
+int sa8d8_rvv(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
+{
+    int cost = 0;
+
+    for (int y = 0; y < h; y += 8)
+        for (int x = 0; x < w; x += 8)
+        {
+            cost += pixel_sa8d_8x8_rvv(pix1 + i_pix1 * y + x, i_pix1, pix2 + i_pix2 * y + x, i_pix2);
+        }
+
+    return cost;
+}
+
+// Calculate sa8d in blocks of 16x16
+template<int w, int h>
+int sa8d16_rvv(const pixel *pix1, intptr_t i_pix1, const pixel *pix2, intptr_t i_pix2)
+{
+    int cost = 0;
+
+    for (int y = 0; y < h; y += 16)
+        for (int x = 0; x < w; x += 16)
+        {
+            cost += pixel_sa8d_16x16_rvv(pix1 + i_pix1 * y + x, i_pix1, pix2 + i_pix2 * y + x, i_pix2);
+        }
+
+    return cost;
+}
+#endif
+
+#if HIGH_BIT_DEPTH
+// todo
+#else // !HIGH_BIT_DEPTH
+static inline int calc_energy_8x8(const uint8_t *source, intptr_t sstride)
+{
+    const uint8_t *pix_ptr = source;
+
+    vuint8mf2_t s0, s1, s2, s3, s4, s5, s6, s7;
+    vectors_u8_mf2_t s_matrix_low = {&s0, &s1, &s2, &s3};
+    vectors_u8_mf2_t s_matrix_high = {&s4, &s5, &s6, &s7};
+
+    size_t vl = __riscv_vsetvl_e8mf2(8);
+    vload_u8x8x4_mf2(&pix_ptr, sstride, &s_matrix_low, vl);
+    pix_ptr += 4 * sstride;
+    vload_u8x8x4_mf2(&pix_ptr, sstride, &s_matrix_high, vl);
+
+    vuint16m1_t diff0, diff1, diff2, diff3, diff4, diff5, diff6, diff7;
+    diff0 = __riscv_vwaddu_vv_u16m1(*(s_matrix_low.m0), *(s_matrix_low.m1), vl);
+    diff1 = __riscv_vwaddu_vv_u16m1(*(s_matrix_low.m2), *(s_matrix_low.m3), vl);
+    diff2 = __riscv_vwaddu_vv_u16m1(*(s_matrix_high.m0), *(s_matrix_high.m1), vl);
+    diff3 = __riscv_vwaddu_vv_u16m1(*(s_matrix_high.m2), *(s_matrix_high.m3), vl);
+    diff4 = __riscv_vwsubu_vv_u16m1(*(s_matrix_low.m0), *(s_matrix_low.m1), vl);
+    diff5 = __riscv_vwsubu_vv_u16m1(*(s_matrix_low.m2), *(s_matrix_low.m3), vl);
+    diff6 = __riscv_vwsubu_vv_u16m1(*(s_matrix_high.m0), *(s_matrix_high.m1), vl);
+    diff7 = __riscv_vwsubu_vv_u16m1(*(s_matrix_high.m2), *(s_matrix_high.m3), vl);
+
+    vectors_u16_m1_t diff_0 = {&diff0, &diff1, &diff2, &diff3};
+    vectors_u16_m1_t diff_1 = {&diff4, &diff5, &diff6, &diff7};
+
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7;
+    vectors_u16_m1_t tmp_0 = {&tmp0, &tmp1, &tmp2, &tmp3};
+    vectors_u16_m1_t tmp_1 = {&tmp4, &tmp5, &tmp6, &tmp7};
+    hadamard_vx4(&tmp_0, &diff_0, vl);
+    hadamard_vx4(&tmp_1, &diff_1, vl);
+
+    // The first line after the vertical hadamard transform contains the sum of coefficients.
+    int sum = vredsum_u16(*(tmp_0.m0), vl) >> 2;
+
+    vuint16m1_t sum0, sum1, sum2, sum3;
+    vectors_u16_m1_t sum_sa8d={&sum0, &sum1, &sum2, &sum3};
+    vectors_u16_m1_8_t tmp={tmp_0.m0, tmp_0.m1, tmp_0.m2, tmp_0.m3, tmp_1.m0, tmp_1.m1, tmp_1.m2, tmp_1.m3};
+    hadamard_hx8(&sum_sa8d, &tmp, vl);
+
+    vuint16m1_t out, out0, out1;
+    out0 = __riscv_vadd_vv_u16m1(*(sum_sa8d.m0), *(sum_sa8d.m1), vl);
+    out1 = __riscv_vadd_vv_u16m1(*(sum_sa8d.m2), *(sum_sa8d.m3), vl);
+    out = __riscv_vadd_vv_u16m1(out0, out1, vl);
+    int sa8 = (vredsum_u16(out, vl) + 1) >> 1;
+
+    return sa8 - sum;
+}
+
+
+static inline int calc_energy_4x4(const pixel *source, intptr_t sstride)
+{
+    const uint8_t *pix_ptr = source;
+    vuint8mf2_t s0, s1, s2, s3;
+    vectors_u8_mf2_t s = {&s0, &s1, &s2, &s3};
+    size_t vl = __riscv_vsetvl_e8mf2(4);
+    vload_u8mf2x4(&s, &pix_ptr, sstride, vl);
+    vuint16m1_t tmp0, tmp1, tmp2, tmp3;
+    tmp0 = __riscv_vwaddu_vx_u16m1(*(s.m0), 0, vl);
+    tmp1 = __riscv_vwaddu_vx_u16m1(*(s.m1), 0, vl);
+    tmp2 = __riscv_vwaddu_vx_u16m1(*(s.m2), 0, vl);
+    tmp3 = __riscv_vwaddu_vx_u16m1(*(s.m3), 0, vl);
+
+    vuint16m1_t s01_23, d01_23, d0, d1;
+    d0 = __riscv_vslideup_vx_u16m1(tmp0, tmp1, 4, 8);
+    d1 = __riscv_vslideup_vx_u16m1(tmp2, tmp3, 4, 8);
+
+    vl = __riscv_vsetvl_e16m1(8);
+    SUMSUB_AB(s01_23, d01_23, d0, d1, vl);
+
+    // The first line after the vertical hadamard transform contains the sum of coefficients.
+    int sum = vredsum_u16(s01_23, vl) >> 2;
+
+    vtrn_16s(&d0, &d1, s01_23, d01_23);
+    SUMSUB_AB(s01_23, d01_23, d0, d1, vl);
+
+    vtrn_8h(&d0, &d1, s01_23, d01_23);
+    SUMSUB_AB(s01_23, d01_23, d0, d1, vl);
+
+    vtrn_4s(&d0, &d1, s01_23, d01_23);
+
+    vuint16m1_t vmax = vmax_abs_u16(d0, d1, vl);
+    int sat = vredsum_u16(vmax, vl);
+
+    return sat - sum;
+}
+
+template<int size>
+int psyCost_pp_rvv(const pixel *source, intptr_t sstride, const pixel *recon, intptr_t rstride)
+{
+    if (size)
+    {
+        int dim = 1 << (size + 2);
+        uint32_t totEnergy = 0;
+        for (int i = 0; i < dim; i += 8)
+        {
+            for (int j = 0; j < dim; j += 8)
+            {
+                int sourceEnergy = calc_energy_8x8(source + i * sstride + j, sstride);
+                int reconEnergy = calc_energy_8x8(recon + i * rstride + j, rstride);
+
+                totEnergy += abs(sourceEnergy - reconEnergy);
+            }
+        }
+        return totEnergy;
+    }
+    else
+    {
+        int sourceEnergy = calc_energy_4x4(source, sstride);
+        int reconEnergy = calc_energy_4x4(recon, rstride);
+
+        return abs(sourceEnergy - reconEnergy);
+    }
+}
+#endif // HIGH_BIT_DEPTH
+
+template<int blockSize>
+void transpose_rvv(pixel *dst, const pixel *src, intptr_t stride)
+{
+    for (int k = 0; k < blockSize; k++)
+        for (int l = 0; l < blockSize; l++)
+        {
+            dstk * blockSize + l = srcl * stride + k;
+        }
+}
+
+template<>
+void transpose_rvv<4>(pixel *dst, const pixel *src, intptr_t stride)
+{
+    transpose4x4_rvv(dst, src, 4, stride);
+}
+
+template<>
+void transpose_rvv<8>(pixel *dst, const pixel *src, intptr_t stride)
+{
+    transpose8x8_rvv(dst, src, 8, stride);
+}
+
+template<>
+void transpose_rvv<16>(pixel *dst, const pixel *src, intptr_t stride)
+{
+    transpose16x16_rvv(dst, src, 16, stride);
+}
+
+template<>
+void transpose_rvv<32>(pixel *dst, const pixel *src, intptr_t stride)
+{
+    transpose32x32_rvv(dst, src, 32, stride);
+}
+
+template<>
+void transpose_rvv<64>(pixel *dst, const pixel *src, intptr_t stride)
+{
+    transpose32x32_rvv(dst, src, 64, stride);
+    transpose32x32_rvv(dst + 32 * 64 + 32, src + 32 * stride + 32, 64, stride);
+    transpose32x32_rvv(dst + 32 * 64, src + 32, 64, stride);
+    transpose32x32_rvv(dst + 32, src + 32 * stride, 64, stride);
+}
+
+};
+
+namespace X265_NS {
+void setupPixelPrimitives_rvv(EncoderPrimitives &p) {
+#define LUMA_PU(W, H) \
+    p.puLUMA_ ## W ## x ## H.pixelavg_ppNONALIGNED = pixelavg_pp_rvv<W, H>; \
+    p.puLUMA_ ## W ## x ## H.pixelavg_ppALIGNED = pixelavg_pp_rvv<W, H>;
+
+#if !(HIGH_BIT_DEPTH)
+#define LUMA_PU_S(W, H)
+#else // (HIGH_BIT_DEPTH)
+#define LUMA_PU_S(W, H) \
+    p.puLUMA_ ## W ## x ## H.pixelavg_ppNONALIGNED = pixelavg_pp_rvv<W, H>; \
+    p.puLUMA_ ## W ## x ## H.pixelavg_ppALIGNED = pixelavg_pp_rvv<W, H>;
+#endif // !(HIGH_BIT_DEPTH)
+
+#define LUMA_CU(W, H) \
+    p.cuBLOCK_ ## W ## x ## H.psy_cost_pp   = psyCost_pp_rvv<BLOCK_ ## W ## x ## H>;
+    //p.cuBLOCK_ ## W ## x ## H.transpose = transpose_rvv<W>;
+
+    LUMA_PU_S(4, 4);
+    LUMA_PU_S(8, 8);
+    LUMA_PU(16, 16);
+    LUMA_PU(32, 32);
+    LUMA_PU(64, 64);
+    LUMA_PU_S(4, 8);
+    LUMA_PU_S(8, 4);
+    LUMA_PU(16,  8);
+    LUMA_PU_S(8, 16);
+    LUMA_PU(16, 12);
+    LUMA_PU(12, 16);
+    LUMA_PU(16,  4);
+    LUMA_PU_S(4, 16);
+    LUMA_PU(32, 16);
+    LUMA_PU(16, 32);
+    LUMA_PU(32, 24);
+    LUMA_PU(24, 32);
+    LUMA_PU(32,  8);
+    LUMA_PU_S(8, 32);
+    LUMA_PU(64, 32);
+    LUMA_PU(32, 64);
+    LUMA_PU(64, 48);
+    LUMA_PU(48, 64);
+    LUMA_PU(64, 16);
+    LUMA_PU(16, 64);
+
+    // transpose
+    p.cuBLOCK_4x4.transpose = transpose_rvv<4>;
+    p.cuBLOCK_8x8.transpose = transpose_rvv<8>;
+    p.cuBLOCK_16x16.transpose = transpose_rvv<16>;
+    p.cuBLOCK_32x32.transpose = transpose_rvv<32>;
+    p.cuBLOCK_64x64.transpose = transpose_rvv<64>;
+
+#if !(HIGH_BIT_DEPTH)
+    LUMA_CU(4, 4);
+    LUMA_CU(8, 8);
+    LUMA_CU(16, 16);
+    LUMA_CU(32, 32);
+    LUMA_CU(64, 64);
+
+    p.puLUMA_4x4.satd = satd4_rvv<4, 4>;
+    p.puLUMA_4x8.satd = satd4_rvv<4, 8>;
+    p.puLUMA_4x16.satd = satd4_rvv<4, 16>;
+    p.puLUMA_8x4.satd = satd8_rvv<8, 4>;
+    p.puLUMA_8x8.satd = satd8_rvv<8, 8>;
+    p.puLUMA_8x16.satd = satd8_rvv<8, 16>;
+    p.puLUMA_8x32.satd = satd8_rvv<8, 32>;
+    p.puLUMA_12x16.satd = satd4_rvv<12, 16>;
+    p.puLUMA_16x4.satd = satd8_rvv<16, 4>;
+    p.puLUMA_16x8.satd = satd8_rvv<16, 8>;
+    p.puLUMA_16x12.satd = satd8_rvv<16, 12>;
+    p.puLUMA_16x16.satd = satd8_rvv<16, 16>;
+    p.puLUMA_16x32.satd = satd8_rvv<16, 32>;
+    p.puLUMA_16x64.satd = satd8_rvv<16, 64>;
+    p.puLUMA_24x32.satd = satd8_rvv<24, 32>;
+    p.puLUMA_32x8.satd = satd8_rvv<32, 8>;
+    p.puLUMA_32x16.satd = satd8_rvv<32, 16>;
+    p.puLUMA_32x24.satd = satd8_rvv<32, 24>;
+    p.puLUMA_32x32.satd = satd8_rvv<32, 32>;
+    p.puLUMA_32x64.satd = satd8_rvv<32, 64>;
+    p.puLUMA_48x64.satd = satd8_rvv<48, 64>;
+    p.puLUMA_64x16.satd = satd8_rvv<64, 16>;
+    p.puLUMA_64x32.satd = satd8_rvv<64, 32>;
+    p.puLUMA_64x48.satd = satd8_rvv<64, 48>;
+    p.puLUMA_64x64.satd = satd8_rvv<64, 64>;
+
+    p.cuBLOCK_4x4.sa8d   = satd4_rvv<4, 4>;
+    p.cuBLOCK_8x8.sa8d   = sa8d8_rvv<8, 8>;
+    p.cuBLOCK_16x16.sa8d = sa8d16_rvv<16, 16>;
+    p.cuBLOCK_32x32.sa8d = sa8d16_rvv<32, 32>;
+    p.cuBLOCK_64x64.sa8d = sa8d16_rvv<64, 64>;
+
+    p.chromaX265_CSP_I420.puCHROMA_420_2x2.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_2x4.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_2x8.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_4x2.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd = satd4_rvv<4, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_4x8.satd = satd4_rvv<4, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_4x16.satd = satd4_rvv<4, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_6x8.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x2.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x4.satd = satd8_rvv<8, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x6.satd = NULL;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x8.satd = satd8_rvv<8, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x16.satd = satd8_rvv<8, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_8x32.satd = satd8_rvv<8, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_12x16.satd = satd4_rvv<12, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x4.satd = satd8_rvv<16, 4>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x8.satd = satd8_rvv<16, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x12.satd = satd8_rvv<16, 12>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x16.satd = satd8_rvv<16, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_16x32.satd = satd8_rvv<16, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_24x32.satd = satd8_rvv<24, 32>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x8.satd = satd8_rvv<32, 8>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x16.satd = satd8_rvv<32, 16>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x24.satd = satd8_rvv<32, 24>;
+    p.chromaX265_CSP_I420.puCHROMA_420_32x32.satd = satd8_rvv<32, 32>;
+
+    p.chromaX265_CSP_I422.puCHROMA_422_2x4.satd = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_2x8.satd = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_2x16.satd = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x4.satd = satd4_rvv<4, 4>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd = satd4_rvv<4, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x16.satd = satd4_rvv<4, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_4x32.satd = satd4_rvv<4, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_6x16.satd = NULL;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x4.satd = satd8_rvv<8, 4>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x8.satd = satd8_rvv<8, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x12.satd = satd8_rvv<8, 12>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x16.satd = satd8_rvv<8, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x32.satd = satd8_rvv<8, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_8x64.satd = satd8_rvv<8, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_12x32.satd = satd4_rvv<12, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x8.satd = satd8_rvv<16, 8>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x16.satd = satd8_rvv<16, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x24.satd = satd8_rvv<16, 24>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x32.satd = satd8_rvv<16, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_16x64.satd = satd8_rvv<16, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_24x64.satd = satd8_rvv<24, 64>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x16.satd = satd8_rvv<32, 16>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x32.satd = satd8_rvv<32, 32>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x48.satd = satd8_rvv<32, 48>;
+    p.chromaX265_CSP_I422.puCHROMA_422_32x64.satd = satd8_rvv<32, 64>;
+
+    p.chromaX265_CSP_I420.cuBLOCK_8x8.sa8d   = p.chromaX265_CSP_I420.puCHROMA_420_4x4.satd;
+    p.chromaX265_CSP_I420.cuBLOCK_16x16.sa8d = sa8d8_rvv<8, 8>;
+    p.chromaX265_CSP_I420.cuBLOCK_32x32.sa8d = sa8d16_rvv<16, 16>;
+    p.chromaX265_CSP_I420.cuBLOCK_64x64.sa8d = sa8d16_rvv<32, 32>;
+
+    p.chromaX265_CSP_I422.cuBLOCK_8x8.sa8d       = p.chromaX265_CSP_I422.puCHROMA_422_4x8.satd;
+    p.chromaX265_CSP_I422.cuBLOCK_16x16.sa8d     = sa8d8_rvv<8, 16>;
+    p.chromaX265_CSP_I422.cuBLOCK_32x32.sa8d     = sa8d16_rvv<16, 32>;
+    p.chromaX265_CSP_I422.cuBLOCK_64x64.sa8d     = sa8d16_rvv<32, 64>;
+
+    p.chromaX265_CSP_I422.cuBLOCK_422_8x16.sa8d  = sa8d8_rvv<8, 16>;
+    p.chromaX265_CSP_I422.cuBLOCK_422_16x32.sa8d = sa8d16_rvv<16, 32>;
+    p.chromaX265_CSP_I422.cuBLOCK_422_32x64.sa8d = sa8d16_rvv<32, 64>;
+
+#else
+    (void)p;
+#endif
+}
+
+} // namespace X265_NS

x265-4.2.tar/source/common/riscv64/pixel-util.S Added

@@ -0,0 +1,3375 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *          Yujiao He <he.yujiao@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+.option arch, +zbb
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+.text
+
+.macro LOAD_4_LINE src, es, stride, d1, d2, d3, d4
+    slli            t2, \stride, 1
+    add             t3, t2, \stride
+    add             t1, \src, \stride
+    add             t2, \src, t2
+    add             t3, \src, t3
+    vl\es\().v      \d1, (\src)
+    vl\es\().v      \d2, (t1)
+    vl\es\().v      \d3, (t2)
+    vl\es\().v      \d4, (t3)
+.endm
+
+#if !HIGH_BIT_DEPTH
+.macro LOAD_DIFF_8x8
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v24, v1, v2, v3
+    add             t0, t3, a1
+    LOAD_4_LINE     t0, e8, a1, v4, v5, v6, v7
+    LOAD_4_LINE     a2, e8, a3, v8, v9, v10, v11
+    add             t0, t3, a3
+    LOAD_4_LINE     t0, e8, a3, v12, v13, v14, v15
+
+    // diff v16~v23
+    vwsubu.vv       v16, v24, v8
+    vwsubu.vv       v17, v1, v9
+    vwsubu.vv       v18, v2, v10
+    vwsubu.vv       v19, v3, v11
+    vwsubu.vv       v20, v4, v12
+    vwsubu.vv       v21, v5, v13
+    vwsubu.vv       v22, v6, v14
+    vwsubu.vv       v23, v7, v15
+.endm
+
+.macro MAX_ABS s0, s1, s2, s3, t0, t1, t2, t3
+    VABS            \s0, \s0, \t0
+    VABS            \s1, \s1, \t1
+    VABS            \s2, \s2, \t2
+    VABS            \s3, \s3, \t3
+
+    vmax.vv         \t0, \s0, \s1
+    vmax.vv         \t1, \s2, \s3
+.endm
+
+.macro SA8D_8x8 dst, s0, s1, s2, s3, s4, s5, s6, s7, t0, t1, t2, t3, t4, t5, t6, t7
+    // v1~3 v8~v15
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD4       \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+    HADAMARD4       \s4, \s5, \s6, \s7, \t0, \t1, \t2, \t3
+    SUMSUB_ABCD     \t0, \t4, \t1, \t5, \s0, \s4, \s1, \s5
+    SUMSUB_ABCD     \t2, \t6, \t3, \t7, \s2, \s6, \s3, \s7
+
+    // transpose v8~v15
+    TRANSPOSE_8x8   16, \s0, \s1, \s2, \s3, \s4, \s5, \s6, \s7, \t0, \t1, \t2, \t3, \t4, \t5, \t6, \t7
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // H1~2
+    HADAMARD4       \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+    HADAMARD4       \s4, \s5, \s6, \s7, \t0, \t1, \t2, \t3
+
+    VABS            \s0, \s0, \t0
+    VABS            \s1, \s1, \t1
+    VABS            \s2, \s2, \t2
+    VABS            \s3, \s3, \t3
+    VABS            \s4, \s4, \t4
+    VABS            \s5, \s5, \t5
+    VABS            \s6, \s6, \t6
+    VABS            \s7, \s7, \t7
+
+    vmv.v.i         \t0, 0
+    vmax.vv         \s0, \s0, \s4
+    vmax.vv         \s1, \s1, \s5
+    vmax.vv         \s2, \s2, \s6
+    vmax.vv         \s3, \s3, \s7
+    vadd.vv         \s0, \s0, \s1
+    vadd.vv         \s2, \s2, \s3
+    vadd.vv         \s0, \s0, \s2
+
+    vwredsumu.vs    \s1, \s0, \t0
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         \dst, \s1
+.endm
+
+//int sa8d_8x8(const pixel* pix1, intptr_t i_pix1, const pixel* pix2, intptr_t i_pix2)
+function PFX(sa8d_8x8_rvv)
+    li              t0, 0x55
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t0
+
+    LOAD_DIFF_8x8
+    SA8D_8x8 a0, v16, v17, v18, v19, v20, v21, v22, v23, v8, v9, v10, v11, v12, v13, v14, v15
+    addi            a0, a0, 1
+    srli            a0, a0, 1
+    ret
+endfunc
+
+//int sa8d_8x16(const pixel* pix1, intptr_t i_pix1, const pixel* pix2, intptr_t i_pix2)
+function PFX(sa8d_8x16_rvv)
+    li              t0, 0x55
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v0, t0
+
+    LOAD_DIFF_8x8
+    SA8D_8x8 a7, v16, v17, v18, v19, v20, v21, v22, v23, v8, v9, v10, v11, v12, v13, v14, v15
+    addi            a7, a7, 1
+    srli            a7, a7, 1
+
+    slli            t0, a1, 3
+    slli            t1, a3, 3
+    add             a0, a0, t0
+    add             a2, a2, t1
+    LOAD_DIFF_8x8
+    SA8D_8x8 a0, v16, v17, v18, v19, v20, v21, v22, v23, v8, v9, v10, v11, v12, v13, v14, v15
+    addi            a0, a0, 1
+    srli            a0, a0, 1
+    add             a0, a0, a7
+
+    ret
+endfunc
+
+.macro LOAD_DIFF_8x16
+    LOAD_4_LINE     a0, e8, a1, v28, v29, v30, v31
+    add             t0, t3, a1
+    LOAD_4_LINE     t0, e8, a1, v4, v5, v6, v7
+    LOAD_4_LINE     a2, e8, a3, v8, v9, v10, v11
+    add             t0, t3, a3
+    LOAD_4_LINE     t0, e8, a3, v12, v13, v14, v15
+
+    // diff v16~v23
+    vwsubu.vv       v16, v28, v8
+    vwsubu.vv       v18, v29, v9
+    vwsubu.vv       v20, v30, v10
+    vwsubu.vv       v22, v31, v11
+    vwsubu.vv       v24, v4, v12
+    vwsubu.vv       v26, v5, v13
+    vwsubu.vv       v28, v6, v14
+    vwsubu.vv       v30, v7, v15
+
+    li              t1, 8
+    vsetivli        t0, 16, e16, m1, ta, ma
+    beq             t0, t1, 1f
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vslidedown.vi   v17, v16, 8
+    vslidedown.vi   v19, v18, 8
+    vslidedown.vi   v21, v20, 8
+    vslidedown.vi   v23, v22, 8
+    vslidedown.vi   v25, v24, 8
+    vslidedown.vi   v27, v26, 8
+    vslidedown.vi   v29, v28, 8
+    vslidedown.vi   v31, v30, 8
+1:
+.endm
+
+.macro SA8D_16x16 dst, stride1, stride2
+    li              t0, 0x55
+    vsetivli        zero, 16, e8, m1, ta, ma
+    vmv.v.x         v0, t0
+
+    LOAD_DIFF_8x16
+    SA8D_8x8 a6, v16, v18, v20, v22, v24, v26, v28, v30, v8, v9, v10, v11, v12, v13, v14, v15
+    SA8D_8x8 t4, v17, v19, v21, v23, v25, v27, v29, v31, v8, v9, v10, v11, v12, v13, v14, v15
+    add             a6, a6, t4
+
+    add             a0, a0, \stride1
+    add             a2, a2, \stride2
+    vsetivli        zero, 16, e8, m1, ta, ma
+    LOAD_DIFF_8x16
+    SA8D_8x8 t4, v16, v18, v20, v22, v24, v26, v28, v30, v8, v9, v10, v11, v12, v13, v14, v15
+    add             a6, a6, t4
+    SA8D_8x8 t4, v17, v19, v21, v23, v25, v27, v29, v31, v8, v9, v10, v11, v12, v13, v14, v15
+    add             a6, a6, t4
+
+    addi            a6, a6, 1
+    srli            \dst, a6, 1
+.endm
+
+//int sa8d_16x16(const pixel* pix1, intptr_t i_pix1, const pixel* pix2, intptr_t i_pix2)
+function PFX(sa8d_16x16_rvv)
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+
+    SA8D_16x16 a0, a4, a5
+    ret
+endfunc
+
+function PFX(sa8d_16x32_rvv)
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+
+    SA8D_16x16 a7, a4, a5
+
+    add             a0, a0, a4
+    add             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+
+    add             a0, a6, a7
+    ret
+endfunc
+
+function PFX(sa8d_32x32_rvv)
+    li              a7, 0
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    slli            t5, a4, 1
+    slli            t6, a5, 1
+
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+    add             a0, a0, a4
+    add             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+    sub             a0, a0, a4
+    sub             a2, a2, a5
+    addi            a0, a0, 16
+    addi            a2, a2, 16
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+    sub             a0, a0, t5
+    sub             a2, a2, t6
+    sub             a0, a0, a4
+    sub             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+    add             a0, a7, a6
+
+    ret
+endfunc
+
+function PFX(sa8d_32x64_rvv)
+    li              a7, 0
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    li              t0, 7
+    mul             t5, a4, t0
+    mul             t6, a5, t0
+
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+.rept 3
+    add             a0, a0, a4
+    add             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+.endr
+
+    addi            a0, a0, 16
+    addi            a2, a2, 16
+    sub             a0, a0, t5
+    sub             a2, a2, t6
+
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+.rept 3
+    add             a0, a0, a4
+    add             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+.endr
+
+    mv              a0, a7
+    ret
+endfunc
+
+function PFX(sa8d_64x64_rvv)
+    li              a7, 0
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    li              t0, 7
+    mul             t5, a4, t0
+    mul             t6, a5, t0
+
+.rept 4
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+
+.rept 3
+    add             a0, a0, a4
+    add             a2, a2, a5
+    SA8D_16x16 a6, a4, a5
+    add             a7, a7, a6
+.endr
+    addi            a0, a0, 16
+    addi            a2, a2, 16
+    sub             a0, a0, t5
+    sub             a2, a2, t6
+.endr
+
+    mv              a0, a7
+    ret
+endfunc
+
+//int psyCost_pp_neon(const pixel *source, intptr_t sstride, const pixel *recon, intptr_t rstride)
+function PFX(psyCost_pp_4x4_rvv)
+    li              t5, 0x55
+    vsetivli        zero, 4, e8, mf4, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v0, v1, v2, v3
+    LOAD_4_LINE     a2, e8, a3, v4, v5, v6, v7
+
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    vslideup.vi     v0, v1, 4
+    vslideup.vi     v2, v3, 4
+    vslideup.vi     v4, v5, 4
+    vslideup.vi     v6, v7, 4
+
+    vwaddu.vv       v8, v0, v2
+    vwsubu.vv       v9, v0, v2
+    vwaddu.vv       v10, v4, v6
+    vwsubu.vv       v11, v4, v6
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+    vredsum.vs      v2, v8, v30
+    vredsum.vs      v3, v10, v30
+    vmv.x.s         t1, v2
+    vmv.x.s         t2, v3
+    srai            t1, t1, 2
+    srai            t2, t2, 2
+
+    vsetivli        zero, 2, e64, m1, ta, ma
+    TRN_4REG        v4, v5, v6, v7, v8, v9, v10, v11
+    vsetivli        zero, 8, e16, m1, ta, ma
+    SUMSUB_ABCD     v8, v9, v10, v11, v4, v5, v6, v7
+    TRN_4REG        v4, v5, v6, v7, v8, v9, v10, v11
+    SUMSUB_ABCD     v8, v9, v10, v11, v4, v5, v6, v7
+    vsetivli        zero, 4, e32, m1, ta, ma
+    TRN_4REG        v4, v5, v6, v7, v8, v9, v10, v11
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    MAX_ABS         v4, v5, v6, v7, v8, v9, v10, v11
+    vwredsumu.vs    v4, v8, v30
+    vwredsumu.vs    v5, v9, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         t3, v4
+    vmv.x.s         t4, v5
+
+    sub             t3, t3, t1
+    sub             t4, t4, t2
+    max             t0, t3, t4
+    min             t1, t3, t4
+    sub             a0, t0, t1
+    ret
+endfunc
+
+// int calc_energy_8x8(const uint8_t *source, intptr_t sstride)
+.macro CALC_ENERGY_8x8 src, sstride, dst
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    LOAD_4_LINE     \src, e8, \sstride, v8, v9, v10, v11
+    add             t0, t3, \sstride
+    LOAD_4_LINE     t0, e8, \sstride, v12, v13, v14, v15
+
+    // diff v16~v23
+    vwaddu.vv       v16, v8, v9
+    vwaddu.vv       v17, v10, v11
+    vwaddu.vv       v18, v12, v13
+    vwaddu.vv       v19, v14, v15
+    vwsubu.vv       v20, v8, v9
+    vwsubu.vv       v21, v10, v11
+    vwsubu.vv       v22, v12, v13
+    vwsubu.vv       v23, v14, v15
+
+    // v1~3 v8~v15
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD4       v16, v17, v18, v19, v8, v9, v10, v11
+    HADAMARD4       v20, v21, v22, v23, v8, v9, v10, v11
+
+    // sum
+    vredsum.vs      v1, v16, v30
+    vmv.x.s         t1, v1
+    srai            t1, t1, 2
+
+    // transpose v8~v15
+    TRANSPOSE_8x8   16, v8, v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20, v21, v22, v23
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    // H1~2
+    HADAMARD4       v8, v9, v10, v11, v16, v17, v18, v19
+    HADAMARD4       v12, v13, v14, v15, v16, v17, v18, v19
+
+    VABS            v8, v8, v16
+    VABS            v9, v9, v17
+    VABS            v10, v10, v18
+    VABS            v11, v11, v19
+    VABS            v12, v12, v20
+    VABS            v13, v13, v21
+    VABS            v14, v14, v22
+    VABS            v15, v15, v23
+
+    vmax.vv         v8, v8, v12
+    vmax.vv         v9, v9, v13
+    vmax.vv         v10, v10, v14
+    vmax.vv         v11, v11, v15
+    vadd.vv         v8, v8, v9
+    vadd.vv         v10, v10, v11
+    vadd.vv         v8, v8, v10
+
+    vwredsumu.vs    v9, v8, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         t2, v9
+    addi            t2, t2, 1
+    srli            t2, t2, 1
+
+    sub             \dst, t2, t1
+.endm
+
+//int psyCost_pp_neon(const pixel *source, intptr_t sstride, const pixel *recon, intptr_t rstride)
+function PFX(psyCost_pp_8x8_rvv)
+    li              t0, 0x55
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v0, t0
+    CALC_ENERGY_8x8 a0, a1, a4
+    CALC_ENERGY_8x8 a2, a3, a5
+    max             t0, a4, a5
+    min             t1, a4, a5
+    sub             a0, t0, t1
+    ret
+endfunc
+
+.macro PSYCOST_PP size
+function PFX(psyCost_pp_\size\()x\size\()_rvv)
+    li              t5, \size
+    li              a7, 0
+    slli            a6, a1, 3
+    slli            t6, a3, 3
+    addi            a6, a6, -\size
+    addi            t6, t6, -\size
+    li              t0, 0x55
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.i         v30, 0
+    vmv.v.x         v0, t0
+looph_psyCost_pp_\size\()x\size\():
+    li              t4, \size
+loopw_psyCost_pp_\size\()x\size\():
+    CALC_ENERGY_8x8 a0, a1, a4
+    CALC_ENERGY_8x8 a2, a3, a5
+    max             t0, a4, a5
+    min             t1, a4, a5
+    sub             t2, t0, t1
+    add             a7, a7, t2
+
+    addi            a0, a0, 8
+    addi            a2, a2, 8
+    addi            t4, t4, -8
+    bgtz            t4, loopw_psyCost_pp_\size\()x\size\()
+    add             a0, a0, a6
+    add             a2, a2, t6
+    addi            t5, t5, -8
+    bgtz            t5, looph_psyCost_pp_\size\()x\size\()
+
+    mv              a0, a7
+    ret
+endfunc
+.endm
+
+PSYCOST_PP  16
+PSYCOST_PP  32
+PSYCOST_PP  64
+
+// int satd4_neon(const pixel *pix1, intptr_t stride_pix1, const pixel *pix2, intptr_t stride_pix2)
+function PFX(satd4_4x4_rvv)
+    li              t5, 0x55
+    vsetivli        zero, 4, e8, mf4, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v0, v1, v2, v3
+    LOAD_4_LINE     a2, e8, a3, v4, v5, v6, v7
+
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    vslideup.vi     v0, v2, 4
+    vslideup.vi     v1, v3, 4
+    vslideup.vi     v4, v6, 4
+    vslideup.vi     v5, v7, 4
+
+    vwsubu.vv       v8, v0, v4
+    vwsubu.vv       v9, v5, v1
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+    SUMSUB_AB       v2, v3, v8, v9
+
+    vsetivli        zero, 2, e64, m1, ta, ma
+    TRN_2REG        v4, v5, v2, v3
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    SUMSUB_AB       v6, v7, v4, v5
+    TRN_2REG        v8, v9, v6, v7
+    SUMSUB_AB       v2, v3, v8, v9
+
+    vsetivli        zero, 4, e32, m1, ta, ma
+    TRN_2REG        v4, v5, v2, v3
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    VABS            v6, v4, v8
+    VABS            v7, v5, v9
+    vmaxu.vv        v2, v6, v7
+    vwredsumu.vs    v1, v2, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v1
+    ret
+endfunc
+
+.macro HADAMARD_4x4_DUAL s0, s1, s2, s3, t0, t1, t2, t3
+    HADAMARD4   \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+
+    TRN_4REG    \t0, \t1, \t2, \t3, \s0, \s1, \s2, \s3
+
+    SUMSUB_ABCD \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+
+    vsetivli    zero, 4, e32, m1, ta, ma
+    TRN_4REG    \t0, \t1, \t2, \t3, \s0, \s2, \s1, \s3
+    vsetivli    zero, 8, e16, m1, ta, ma
+
+    MAX_ABS     \t0, \t1, \t2, \t3, \s0, \s1, \s2, \s3
+    vadd.vv     \s0, \s0, \s1
+.endm
+
+.macro SATD4_4x8 dst, src1, stride1, src2, stride2, v0, v30
+    vsetivli        zero, 4, e8, mf4, ta, ma
+    LOAD_4_LINE     \src1, e8, \stride1, v20, v1, v2, v3
+    add             t0, t3, \stride1
+    LOAD_4_LINE     t0, e8, \stride1, v4, v5, v6, v7
+    LOAD_4_LINE     \src2, e8, \stride2, v8, v9, v10, v11
+    add             t0, t3, \stride2
+    LOAD_4_LINE     t0, e8, \stride2, v12, v13, v14, v15
+
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    vslideup.vi     v20, v4, 4
+    vslideup.vi     v1, v5, 4
+    vslideup.vi     v2, v6, 4
+    vslideup.vi     v3, v7, 4
+    vslideup.vi     v8, v12, 4
+    vslideup.vi     v9, v13, 4
+    vslideup.vi     v10, v14, 4
+    vslideup.vi     v11, v15, 4
+
+    vwsubu.vv       v4, v20, v8
+    vwsubu.vv       v5, v9, v1
+    vwsubu.vv       v6, v2, v10
+    vwsubu.vv       v7, v11, v3
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD_4x4_DUAL v4, v5, v6, v7, v8, v9, v10, v11
+    vwredsumu.vs    v1, v4, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         \dst, v1
+.endm
+
+function PFX(satd4_4x8_rvv)
+    li              t5, 0x55
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+    SATD4_4x8       a0, a0, a1, a2, a3, v0, v30
+    ret
+endfunc
+
+.macro SATD4_4xN N
+function PFX(satd4_4x\N\()_rvv)
+    li              t5, 0x55
+    li              t4, \N
+    li              a7, 0
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+
+loop_satd4_4x\N:
+    SATD4_4x8       a6, a0, a1, a2, a3, v0, v30
+    add             a7, a7, a6
+    addi            t4, t4, -8
+    add             a0, a0, a4
+    add             a2, a2, a5
+    bgtz            t4, loop_satd4_4x\N
+
+    mv              a0, a7
+    ret
+endfunc
+.endm
+
+SATD4_4xN 16
+SATD4_4xN 32
+
+.macro SATD4_WxH W, H
+function PFX(satd4_\W\()x\H\()_rvv)
+    li              t5, 0x55
+    li              t4, \H
+    li              a7, 0
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    addi            a4, a4, -\W
+    addi            a5, a5, -\W
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+
+looph_satd4_\W\()x\H:
+    li              t6, \W
+loopw_satd4_\W\()x\H:
+    SATD4_4x8       a6, a0, a1, a2, a3, v0, v30
+    add             a7, a7, a6
+    addi            a0, a0, 4
+    addi            a2, a2, 4
+    addi            t6, t6, -4
+    bgtz            t6, loopw_satd4_\W\()x\H
+    add             a0, a0, a4
+    add             a2, a2, a5
+    addi            t4, t4, -8
+    bgtz            t4, looph_satd4_\W\()x\H
+
+    mv              a0, a7
+    ret
+endfunc
+.endm
+
+SATD4_WxH 12, 16
+SATD4_WxH 12, 32
+
+// int pixel_satd_16x16_neon(const uint8_t *pix1, intptr_t stride_pix1, const uint8_t *pix2, intptr_t stride_pix2)
+function PFX(satd8_8x4_rvv)
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v1, v2, v3, v4
+    LOAD_4_LINE     a2, e8, a3, v5, v6, v7, v8
+
+    vwsubu.vv       v10, v1, v5
+    vwsubu.vv       v11, v2, v6
+    vwsubu.vv       v12, v3, v7
+    vwsubu.vv       v13, v4, v8
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    li              t5, 0x55
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+    HADAMARD_4x4_DUAL v10, v11, v12, v13, v14, v15, v16, v17
+    vwredsumu.vs    v1, v10, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v1
+
+    ret
+endfunc
+
+function PFX(satd8_8x12_rvv)
+    li              t5, 0x55
+    li              t4, 12
+    slli            a4, a1, 2
+    slli            a5, a3, 2
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+    vmv.v.i         v31, 0
+
+loop_satd8_8x12:
+    vsetivli        zero, 8, e8, mf2, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v1, v2, v3, v4
+    LOAD_4_LINE     a2, e8, a3, v5, v6, v7, v8
+
+    vwsubu.vv       v10, v1, v5
+    vwsubu.vv       v11, v2, v6
+    vwsubu.vv       v12, v3, v7
+    vwsubu.vv       v13, v4, v8
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD_4x4_DUAL v10, v11, v12, v13, v14, v15, v16, v17
+    vadd.vv         v31, v31, v10
+
+    add             a0, a0, a4
+    add             a2, a2, a5
+    addi            t4, t4, -4
+    bgtz            t4, loop_satd8_8x12
+
+    vwredsumu.vs    v1, v31, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v1
+
+    ret
+endfunc
+
+.macro HADAMARD_4x4_QUAD s0, s1, s2, s3, s4, s5, s6, s7, t0, t1, t2, t3, t4, t5, t6, t7
+    HADAMARD4       \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+    HADAMARD4       \s4, \s5, \s6, \s7, \t0, \t1, \t2, \t3
+
+    TRN_8REG        \t0, \t1, \t2, \t3, \t4, \t5, \t6, \t7, \s0, \s1, \s2, \s3, \s4, \s5, \s6, \s7
+    SUMSUB_ABCD     \s0, \s1, \s2, \s3, \t0, \t1, \t2, \t3
+    SUMSUB_ABCD     \s4, \s5, \s6, \s7, \t4, \t5, \t6, \t7
+
+    vsetivli        zero, 4, e32, m1, ta, ma
+    TRN_8REG        \t0, \t1, \t2, \t3, \t4, \t5, \t6, \t7, \s0, \s2, \s1, \s3, \s4, \s6, \s5, \s7
+    vsetivli        zero, 8, e16, m1, ta, ma
+
+    MAX_ABS         \t0, \t1, \t2, \t3, \s0, \s1, \s2, \s3
+    MAX_ABS         \t4, \t5, \t6, \t7, \s4, \s5, \s6, \s7
+
+    vadd.vv         \s0, \s0, \s1
+    vadd.vv         \s4, \s4, \s5
+    vadd.vv         \s0, \s0, \s4
+.endm
+
+function PFX(satd8_8x8_rvv)
+    LOAD_DIFF_8x8
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    li              t5, 0x55
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+
+    HADAMARD_4x4_QUAD v16, v17, v18, v19, v20, v21, v22, v23, v8, v9, v10, v11, v12, v13, v14, v15
+
+    vwredsumu.vs    v2, v16, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v2
+    ret
+endfunc
+
+.macro SATD8S_WxH W, H
+function PFX(satd8_\W\()x\H\()_rvv)
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vmv.v.i         v30, 0
+    vsetivli        zero, 8, e16, m1, ta, ma
+    li              t5, 0x55
+    vmv.v.x         v0, t5
+    vmv.v.i         v28, 0
+
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    addi            a4, a4, -\W
+    addi            a5, a5, -\W
+    li              t4, \H
+looph_satd8_\W\()x\H\():
+.if \W > 8
+    li              t5, \W
+loopw_satd8_\W\()x\H\():
+.endif
+
+    LOAD_DIFF_8x8
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD_4x4_QUAD v16, v17, v18, v19, v20, v21, v22, v23, v8, v9, v10, v11, v12, v13, v14, v15
+    vwaddu.wv       v30, v30, v16
+
+    addi            a0, a0, 8
+    addi            a2, a2, 8
+.if \W > 8
+    addi            t5, t5, -8
+    bgtz            t5, loopw_satd8_\W\()x\H
+.endif
+    addi            t4, t4, -8
+    add             a0, a0, a4
+    add             a2, a2, a5
+    bgtz            t4, looph_satd8_\W\()x\H
+
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vredsum.vs      v2, v30, v28
+    vmv.x.s         a0, v2
+    ret
+endfunc
+.endm
+
+SATD8S_WxH 8, 16
+SATD8S_WxH 8, 32
+SATD8S_WxH 8, 64
+SATD8S_WxH 24, 32
+SATD8S_WxH 24, 64
+
+function PFX(satd8_16x4_rvv)
+    li              t5, 0x55
+    vsetivli        zero, 16, e8, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+
+    LOAD_4_LINE     a0, e8, a1, v8, v9, v10, v11
+    LOAD_4_LINE     a2, e8, a3, v12, v13, v14, v15
+
+    vwsubu.vv       v16, v8, v12
+    vwsubu.vv       v18, v9, v13
+    vwsubu.vv       v20, v10, v14
+    vwsubu.vv       v22, v11, v15
+
+    li              t0, 8
+    vsetivli        t1, 16, e16, m1, ta, ma
+    beq             t0, t1, 1f
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vslidedown.vi   v17, v16, 8
+    vslidedown.vi   v19, v18, 8
+    vslidedown.vi   v21, v20, 8
+    vslidedown.vi   v23, v22, 8
+
+1:
+    HADAMARD_4x4_DUAL v16, v18, v20, v22, v8, v9, v10, v11
+    HADAMARD_4x4_DUAL v17, v19, v21, v23, v8, v9, v10, v11
+
+    vadd.vv         v16, v16, v17
+    vwredsumu.vs    v8, v16, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v8
+
+    ret
+endfunc
+
+function PFX(satd8_16x12_rvv)
+    li              t5, 0x55
+    li              t6, 3
+    slli            a4, a1, 2
+    slli            a5, a3, 2
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vmv.v.i         v28, 0
+    vsetivli        zero, 16, e8, m1, ta, ma
+    vmv.v.x         v0, t5
+    vmv.v.i         v30, 0
+
+loop_satd8_16x12:
+    vsetivli        zero, 16, e8, m1, ta, ma
+    LOAD_4_LINE     a0, e8, a1, v8, v9, v10, v11
+    LOAD_4_LINE     a2, e8, a3, v12, v13, v14, v15
+
+    vwsubu.vv       v16, v8, v12
+    vwsubu.vv       v18, v9, v13
+    vwsubu.vv       v20, v10, v14
+    vwsubu.vv       v22, v11, v15
+
+    li              t0, 8
+    vsetivli        t1, 16, e16, m1, ta, ma
+    beq             t0, t1, 1f
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vslidedown.vi   v17, v16, 8
+    vslidedown.vi   v19, v18, 8
+    vslidedown.vi   v21, v20, 8
+    vslidedown.vi   v23, v22, 8
+
+1:
+    HADAMARD_4x4_DUAL v16, v18, v20, v22, v8, v9, v10, v11
+    HADAMARD_4x4_DUAL v17, v19, v21, v23, v8, v9, v10, v11
+    vadd.vv         v16, v16, v17
+    vwaddu.wv       v28, v28, v16
+
+    add             a0, a0, a4
+    add             a2, a2, a5
+    addi            t6, t6, -1
+    bgtz            t6, loop_satd8_16x12
+
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vredsum.vs      v8, v28, v30
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v8
+
+    ret
+endfunc
+
+.macro SATD8_16x8 v0, v2
+    vsetivli        zero, 16, e8, m1, ta, ma
+    LOAD_DIFF_8x16
+
+    vsetivli        zero, 8, e16, m1, ta, ma
+    HADAMARD_4x4_QUAD v16, v18, v20, v22, v24, v26, v28, v30, v8, v9, v10, v11, v12, v13, v14, v15
+    HADAMARD_4x4_QUAD v17, v19, v21, v23, v25, v27, v29, v31, v8, v9, v10, v11, v12, v13, v14, v15
+
+    vadd.vv         v16, v16, v17
+    vwaddu.wv       \v2, \v2, v16
+.endm
+
+.macro SATD8_WxH W, H
+function PFX(satd8_\W\()x\H\()_rvv)
+    li              t5, 0x55
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vmv.v.i         v2, 0
+    vsetivli        zero, 16, e8, m1, ta, ma
+    vmv.v.i         v1, 0
+    vmv.v.x         v0, t5
+
+.if \H > 8
+    slli            a4, a1, 3
+    slli            a5, a3, 3
+    addi            a4, a4, -\W
+    addi            a5, a5, -\W
+    li              t4, \H
+looph_satd8_\W\()x\H:
+.endif
+.if \W > 16
+    li              t5, \W
+loopw_satd8_\W\()x\H:
+.endif
+
+    SATD8_16x8      v0, v2
+
+    addi            a0, a0, 16
+    addi            a2, a2, 16
+.if \W > 16
+    addi            t5, t5, -16
+    bgtz            t5, loopw_satd8_\W\()x\H
+.endif
+.if \H > 8
+    add             a0, a0, a4
+    add             a2, a2, a5
+    addi            t4, t4, -8
+    bgtz            t4, looph_satd8_\W\()x\H
+.endif
+
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vredsum.vs      v8, v2, v1
+    vsetivli        zero, 1, e32, m1, ta, ma
+    vmv.x.s         a0, v8
+    ret
+endfunc
+.endm
+
+SATD8_WxH 16, 8
+SATD8_WxH 16, 16
+SATD8_WxH 16, 24
+SATD8_WxH 16, 32
+SATD8_WxH 16, 64
+SATD8_WxH 32, 8
+SATD8_WxH 32, 16
+SATD8_WxH 32, 24
+SATD8_WxH 32, 32
+SATD8_WxH 32, 48
+SATD8_WxH 32, 64
+SATD8_WxH 48, 64
+SATD8_WxH 64, 16
+SATD8_WxH 64, 32
+SATD8_WxH 64, 48
+SATD8_WxH 64, 64
+
+#endif
+
+// uint64_t pixel_var(const pixel* pix, intptr_t i_stride)
+.macro PIXEL_VAR size, lmul, lmul2, lmul4
+function PFX(pixel_var_\size\()x\size\()_v)
+    li              t1, \size
+    li              t3, \size
+#if HIGH_BIT_DEPTH
+    slli            t2, a1, 1
+#else
+    mv              t2, a1
+#endif
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+
+loop_pixelvar\size:
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t3, e16, m\lmul2, ta, ma
+    vle16.v         v0, (a0)
+    vwaddu.wv       v8, v8, v0
+    vwmulu.vv       v24, v0, v0
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vadd.vv         v16, v16, v24
+#else
+    vsetvli         zero, t3, e8, m\lmul, ta, ma
+    vle8.v          v0, (a0)
+    vwmulu.vv       v4, v0, v0
+    vsetvli         zero, t3, e16, m\lmul2, ta, ma
+    vzext.vf2       v24, v0
+    vwaddu.wv       v8, v8, v24
+    vwaddu.wv       v16, v16, v4
+#endif
+    addi            t1, t1, -1
+    add             a0, a0, t2
+    bgtz            t1, loop_pixelvar\size
+
+    vsetvli         zero, t3, e32, m1, ta, ma
+    vmv.v.i         v0, 0
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vredsum.vs      v1, v8, v0
+    vredsum.vs      v2, v16, v0
+    vmv.x.s         t1, v1
+    vmv.x.s         t2, v2
+    slli            t2, t2, 32
+    add             a0, t1, t2
+    ret
+endfunc
+.endm
+
+.macro PIXEL_VAR2 size, lmul, lmul2, lmul4
+function PFX(pixel_var_\size\()x\size\()_v)
+    li              t1, \size
+    li              t3, \size
+    slli            a2, a1, 1
+    slli            a3, a2, 1
+    add             a4, a3, a2
+    add             a5, a2, a1
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+
+loop_pixelvar\size:
+#if HIGH_BIT_DEPTH
+    add             t4, a0, a2
+    add             t5, a0, a3
+    add             t6, a0, a4
+    vsetvli         zero, t3, e16, m\lmul2, ta, ma
+    vle16.v         v0, (a0)
+    vle16.v         v2, (t4)
+    vle16.v         v4, (t5)
+    vle16.v         v6, (t6)
+    add             a0, t6, a2
+    vwaddu.wv       v8, v8, v0
+    vwaddu.wv       v8, v8, v2
+    vwaddu.wv       v8, v8, v4
+    vwaddu.wv       v8, v8, v6
+    vwmulu.vv       v24, v0, v0
+    vwmulu.vv       v12, v2, v2
+    vwmulu.vv       v20, v4, v4
+    vwmulu.vv       v28, v6, v6
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vadd.vv         v16, v16, v24
+    vadd.vv         v16, v16, v12
+    vadd.vv         v16, v16, v20
+    vadd.vv         v16, v16, v28
+#else
+    add             t4, a0, a1
+    add             t5, a0, a2
+    add             t6, a0, a5
+    vsetvli         zero, t3, e8, m\lmul, ta, ma
+    vle8.v          v20, (a0)
+    vle8.v          v21, (t4)
+    vle8.v          v22, (t5)
+    vle8.v          v23, (t6)
+    add             a0, t6, a1
+    vwmulu.vv       v24, v20, v20
+    vwmulu.vv       v26, v21, v21
+    vwmulu.vv       v28, v22, v22
+    vwmulu.vv       v30, v23, v23
+    vsetvli         zero, t3, e16, m\lmul2, ta, ma
+    vzext.vf2       v0, v20
+    vzext.vf2       v2, v21
+    vzext.vf2       v4, v22
+    vzext.vf2       v6, v23
+    vwaddu.wv       v8, v8, v0
+    vwaddu.wv       v8, v8, v2
+    vwaddu.wv       v8, v8, v4
+    vwaddu.wv       v8, v8, v6
+    vwaddu.wv       v16, v16, v24
+    vwaddu.wv       v16, v16, v26
+    vwaddu.wv       v16, v16, v28
+    vwaddu.wv       v16, v16, v30
+#endif
+    addi            t1, t1, -4
+    bgtz            t1, loop_pixelvar\size
+
+    vsetvli         zero, t3, e32, m1, ta, ma
+    vmv.v.i         v0, 0
+    vsetvli         zero, t3, e32, m\lmul4, ta, ma
+    vredsum.vs      v1, v8, v0
+    vredsum.vs      v2, v16, v0
+    vmv.x.s         t1, v1
+    vmv.x.s         t2, v2
+    slli            t2, t2, 32
+    add             a0, t1, t2
+    ret
+endfunc
+.endm
+
+PIXEL_VAR2 4, 1, 1, 1
+PIXEL_VAR2 8, 1, 1, 2
+PIXEL_VAR2 16, 1, 2, 4
+PIXEL_VAR  32, 2, 4, 8
+
+// uint64_t pixel_var(const pixel* pix, intptr_t i_stride)
+function PFX(pixel_var_64x64_v)
+    li              t1, 64
+#if HIGH_BIT_DEPTH
+    slli            t2, a1, 1
+#else
+    mv              t2, a1
+#endif
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+
+loop_pixelvar64:
+    li              t3, 64
+    mv              t4, a0
+loop_pixelvar264:
+#if HIGH_BIT_DEPTH
+    vsetvli         t5, t3, e16, m4, ta, ma
+    vle16.v         v0, (t4)
+#else
+    vsetvli         t5, t3, e8, m2, ta, ma
+    vle8.v          v4, (t4)
+    vsetvli         zero, t5, e16, m4, ta, ma
+    vzext.vf2       v0, v4
+#endif
+    vwaddu.wv       v8, v8, v0
+    vwmulu.vv       v24, v0, v0
+    vsetvli         zero, t5, e32, m8, ta, ma
+    vadd.vv         v16, v16, v24
+#if HIGH_BIT_DEPTH
+    slli            t6, t5, 1
+    add             t4, t4, t6
+#else
+    add             t4, t4, t5
+#endif
+    sub             t3, t3, t5
+    bgtz            t3, loop_pixelvar264
+    addi            t1, t1, -1
+    add             a0, a0, t2
+    bgtz            t1, loop_pixelvar64
+
+    li              t1, 64
+    vsetvli         zero, t1, e32, m1, ta, ma
+    vmv.v.i         v0, 0
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vredsum.vs      v1, v8, v0
+    vredsum.vs      v2, v16, v0
+    vmv.x.s         t1, v1
+    vmv.x.s         t2, v2
+    slli            t2, t2, 32
+    add             a0, t1, t2
+    ret
+endfunc
+
+// void getResidual(const pixel* fenc, const pixel* pred, int16_t* residual, intptr_t stride)
+.macro GET_RESIDUAL blocksize, lmul, lmul2
+function PFX(getResidual_\blocksize\()_v)
+    li              t1, \blocksize
+    slli            a4, a3, 1
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t1, e16, m\lmul2, ta, ma
+#else
+    vsetvli         zero, t1, e8, m\lmul, ta, ma
+#endif
+loop_residual\blocksize:
+    add             t4, a2, a4
+#if HIGH_BIT_DEPTH
+    add             t2, a0, a4
+    add             t3, a1, a4
+    vle16.v         v0, (a0)
+    vle16.v         v12, (t2)
+    vle16.v         v4, (a1)
+    vle16.v         v16, (t3)
+    add             a0, t2, a4
+    add             a1, t3, a4
+    vsub.vv         v8, v0, v4
+    vsub.vv         v20, v12, v16
+#else
+    add             t2, a0, a3
+    add             t3, a1, a3
+    vle8.v          v0, (a0)
+    vle8.v          v12, (t2)
+    vle8.v          v4, (a1)
+    vle8.v          v16, (t3)
+    add             a0, t2, a3
+    add             a1, t3, a3
+    vwsubu.vv       v8, v0, v4
+    vwsubu.vv       v20, v12, v16
+#endif
+    vse16.v         v8, (a2)
+    vse16.v         v20, (t4)
+    add             a2, t4, a4
+    addi            t1, t1, -2
+    bgtz            t1, loop_residual\blocksize
+    ret
+endfunc
+.endm
+
+.macro GET_RESIDUAL2 blocksize, lmul, lmul2
+function PFX(getResidual_\blocksize\()_v)
+    li              t1, \blocksize
+    slli            a4, a3, 1
+    slli            a5, a4, 1
+    add             a6, a5, a4
+    add             a7, a4, a3
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t1, e16, m\lmul2, ta, ma
+#else
+    vsetvli         zero, t1, e8, m\lmul, ta, ma
+#endif
+loop_residual\blocksize:
+#if HIGH_BIT_DEPTH
+    add             t2, a0, a4
+    add             t3, a1, a4
+    add             t4, a0, a5
+    add             t5, a1, a5
+    add             t0, a0, a6
+    add             t6, a1, a6
+    vle16.v         v0, (a0)
+    vle16.v         v12, (t2)
+    vle16.v         v2, (t4)
+    vle16.v         v14, (t0)
+    vle16.v         v4, (a1)
+    vle16.v         v16, (t3)
+    vle16.v         v6, (t5)
+    vle16.v         v18, (t6)
+    add             a0, t0, a4
+    add             a1, t6, a4
+    vsub.vv         v8, v0, v4
+    vsub.vv         v20, v12, v16
+    vsub.vv         v10, v2, v6
+    vsub.vv         v22, v14, v18
+#else
+    add             t2, a0, a3
+    add             t3, a1, a3
+    add             t4, a0, a4
+    add             t5, a1, a4
+    add             t0, a0, a7
+    add             t6, a1, a7
+    vle8.v          v0, (a0)
+    vle8.v          v12, (t2)
+    vle8.v          v2, (t4)
+    vle8.v          v14, (t0)
+    vle8.v          v4, (a1)
+    vle8.v          v16, (t3)
+    vle8.v          v6, (t5)
+    vle8.v          v18, (t6)
+    add             a0, t0, a3
+    add             a1, t6, a3
+    vwsubu.vv       v8, v0, v4
+    vwsubu.vv       v20, v12, v16
+    vwsubu.vv       v10, v2, v6
+    vwsubu.vv       v22, v14, v18
+#endif
+    add             t2, a2, a4
+    add             t3, a2, a5
+    add             t4, a2, a6
+    vse16.v         v8, (a2)
+    vse16.v         v20, (t2)
+    vse16.v         v10, (t3)
+    vse16.v         v22, (t4)
+    add             a2, t4, a4
+    addi            t1, t1, -4
+    bgtz            t1, loop_residual\blocksize
+    ret
+endfunc
+.endm
+
+GET_RESIDUAL2 4,  1, 1
+GET_RESIDUAL2 8,  1, 1
+GET_RESIDUAL2 16, 1, 2
+GET_RESIDUAL  32, 2, 4
+
+// void pixel_sub_ps_c(int16_t* a, intptr_t dstride, const pixel* b0, const pixel* b1, intptr_t sstride0, intptr_t sstride1)
+.macro SUB_PS xsize, ysize, lmul, lmul2
+function PFX(pixel_sub_ps_\xsize\()x\ysize\()_v)
+    li              t0, \xsize
+    li              t1, \ysize
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a4, a4, 1
+    slli            a5, a5, 1
+#endif
+loop_subps_\xsize\()x\ysize\():
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    vle16.v         v0, (a2)
+    vle16.v         v8, (a3)
+    vsub.vv         v16, v0, v8
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vle8.v          v0, (a2)
+    vle8.v          v4, (a3)
+    vwsubu.vv       v16, v0, v4
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+#endif
+    vse16.v         v16, (a0)
+    add             a0, a0, a1
+    add             a2, a2, a4
+    add             a3, a3, a5
+    addi            t1, t1, -1
+    bgtz            t1, loop_subps_\xsize\()x\ysize\()
+    ret
+endfunc
+.endm
+
+.macro SUB_PS2 xsize, ysize, lmul, lmul2
+function PFX(pixel_sub_ps_\xsize\()x\ysize\()_v)
+    li              t0, \xsize
+    li              t1, \ysize
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a4, a4, 1
+    slli            a5, a5, 1
+#endif
+loop_subps_\xsize\()x\ysize\():
+    add             t3, a2, a4
+    add             t4, a3, a5
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+    vle16.v         v8, (a3)
+    vle16.v         v12, (t4)
+    vsub.vv         v16, v0, v8
+    vsub.vv         v20, v4, v12
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vle8.v          v0, (a2)
+    vle8.v          v2, (t3)
+    vle8.v          v4, (a3)
+    vle8.v          v6, (t4)
+    vwsubu.vv       v16, v0, v4
+    vwsubu.vv       v20, v2, v6
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+#endif
+    add             a2, t3, a4
+    add             a3, t4, a5
+    add             t5, a0, a1
+    vse16.v         v16, (a0)
+    vse16.v         v20, (t5)
+    add             a0, t5, a1
+    addi            t1, t1, -2
+    bgtz            t1, loop_subps_\xsize\()x\ysize\()
+    ret
+endfunc
+.endm
+
+SUB_PS2  4,  4,  1, 1
+SUB_PS2  4,  8,  1, 1
+SUB_PS2  8,  8,  1, 1
+SUB_PS2  8,  16, 1, 1
+SUB_PS2  16, 16, 1, 2
+SUB_PS2  16, 32, 1, 2
+SUB_PS2  32, 32, 2, 4
+SUB_PS2  32, 64, 2, 4
+SUB_PS   64, 64, 4, 8
+
+// void pixel_add_ps_c(pixel* a, intptr_t dstride, const pixel* b0, const int16_t* b1, intptr_t sstride0, intptr_t sstride1)
+.macro ADD_PS xsize, ysize, lmul, lmul2
+function PFX(pixel_add_ps_\xsize\()x\ysize\()_v)
+    li              t0, \xsize
+    li              t1, \ysize
+    li              t2, (1 << BIT_DEPTH) - 1
+    slli            a5, a5, 1
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a4, a4, 1
+#endif
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    vmv.v.i         v24, 0
+.if \lmul2 <= 4
+    vmv.v.x         v28, t2
+.endif
+loop_addps_\xsize\()x\ysize\():
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    vle16.v         v8, (a3)
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vadd.vv         v16, v0, v8
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vle8.v          v0, (a2)
+    vwaddu.wv       v16, v8, v0
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+#endif
+    vmax.vv         v8, v16, v24
+.if \lmul2 <= 4
+    vminu.vv        v16, v8, v28
+.else
+    vmv.v.x         v0, t2
+    vminu.vv        v16, v8, v0
+.endif
+#if HIGH_BIT_DEPTH
+    vse16.v         v16, (a0)
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vnsrl.wi        v8, v16, 0
+    vse8.v          v8, (a0)
+#endif
+    addi            t1, t1, -1
+    add             a0, a0, a1
+    add             a2, a2, a4
+    add             a3, a3, a5
+    bgtz            t1, loop_addps_\xsize\()x\ysize\()
+    ret
+endfunc
+.endm
+
+.macro ADD_PS2 xsize, ysize, lmul, lmul2
+function PFX(pixel_add_ps_\xsize\()x\ysize\()_v)
+    li              t0, \xsize
+    li              t1, \ysize
+    li              t2, (1 << BIT_DEPTH) - 1
+    slli            a5, a5, 1
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a4, a4, 1
+#endif
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    vmv.v.i         v24, 0
+    vmv.v.x         v28, t2
+loop_addps_\xsize\()x\ysize\():
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+    add             t3, a3, a5
+    add             t4, a2, a4
+    vle16.v         v8, (a3)
+    vle16.v         v12, (t3)
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t4)
+    vadd.vv         v16, v0, v8
+    vadd.vv         v20, v4, v12
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vle8.v          v0, (a2)
+    vle8.v          v4, (t4)
+    vwaddu.wv       v16, v8, v0
+    vwaddu.wv       v20, v12, v4
+    vsetvli         zero, t0, e16, m\lmul2, ta, ma
+#endif
+    add             a2, t4, a4
+    add             a3, t3, a5
+    vmax.vv         v16, v16, v24
+    vmax.vv         v20, v20, v24
+    vminu.vv        v16, v16, v28
+    vminu.vv        v20, v20, v28
+    add             t3, a0, a1
+#if HIGH_BIT_DEPTH
+    vse16.v         v16, (a0)
+    vse16.v         v20, (t3)
+#else
+    vsetvli         zero, t0, e8, m\lmul, ta, ma
+    vnsrl.wi        v0, v16, 0
+    vnsrl.wi        v4, v20, 0
+    vse8.v          v0, (a0)
+    vse8.v          v4, (t3)
+#endif
+    addi            t1, t1, -2
+    add             a0, t3, a1
+    bgtz            t1, loop_addps_\xsize\()x\ysize\()
+    ret
+endfunc
+.endm
+
+ADD_PS2 4,  4,  1, 1
+ADD_PS2 4,  8,  1, 1
+ADD_PS2 8,  8,  1, 1
+ADD_PS2 8,  16, 1, 1
+ADD_PS2 16, 16, 1, 2
+ADD_PS2 16, 32, 1, 2
+ADD_PS2 32, 64, 2, 4
+ADD_PS2 32, 32, 2, 4
+ADD_PS  64, 64, 4, 8
+
+// void scale1D_128to64(pixel *dst, const pixel *src)
+function PFX(scale1D_128to64_v)
+    li              t1, 64
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t1, e16, m8, ta, ma
+    li              t2, 4
+    addi            t3, a1, 2
+    addi            t4, a1, 256
+    addi            t5, a1, 258
+    addi            t6, a0, 128
+    vlse16.v        v0, (a1), t2
+    vlse16.v        v8, (t3), t2
+    vlse16.v        v16, (t4), t2
+    vlse16.v        v24, (t5), t2
+    vaaddu.vv       v0, v0, v8
+    vaaddu.vv       v8, v16, v24
+    vse16.v         v0, (a0)
+    vse16.v         v8, (t6)
+#else
+    vsetvli         zero, t1, e8, m4, ta, ma
+    li              t2, 2
+    addi            t3, a1, 1
+    addi            t4, a1, 128
+    addi            t5, a1, 129
+    addi            t6, a0, 64
+    vlse8.v         v0, (a1), t2
+    vlse8.v         v8, (t3), t2
+    vlse8.v         v16, (t4), t2
+    vlse8.v         v24, (t5), t2
+    vaaddu.vv       v0, v0, v8
+    vaaddu.vv       v8, v16, v24
+    vse8.v          v0, (a0)
+    vse8.v          v8, (t6)
+#endif
+    ret
+endfunc
+
+// void scale2D_64to32(pixel* dst, const pixel* src, intptr_t stride)
+function PFX(scale2D_64to32_v)
+    li              t1, 32
+    li              t2, 32
+
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t1, e16, m4, ta, ma
+    li              t3, 4
+    slli            a2, a2, 1
+    slli            a3, a2, 1
+loop_scale2D_64to32:
+    addi            t4, a1, 2
+    add             t5, a1, a2
+    add             t6, t4, a2
+    vlse16.v        v0, (a1), t3
+    vlse16.v        v4, (t4), t3
+    vlse16.v        v8, (t5), t3
+    vlse16.v        v12, (t6), t3
+    vadd.vv         v0, v0, v4
+    vadd.vv         v0, v0, v8
+    vadd.vv         v0, v0, v12
+    vadd.vi         v0, v0, 2
+    vsrl.vi         v0, v0, 2
+    vse16.v         v0, (a0)
+    addi            a0, a0, 64
+#else
+    vsetvli         zero, t1, e8, m2, ta, ma
+    li              t3, 2
+    slli            a3, a2, 1
+loop_scale2D_64to32:
+    addi            t4, a1, 1
+    add             t5, a1, a2
+    add             t6, t4, a2
+    vlse8.v         v0, (a1), t3
+    vlse8.v         v4, (t4), t3
+    vlse8.v         v8, (t5), t3
+    vlse8.v         v12, (t6), t3
+    vwaddu.vv       v16, v0, v4
+    vwaddu.vv       v20, v8, v12
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vadd.vv         v4, v16, v20
+    vadd.vi         v4, v4, 2
+    vsetvli         zero, t1, e8, m2, ta, ma
+    vnsrl.wi        v0, v4, 2
+    vse8.v          v0, (a0)
+    addi            a0, a0, 32
+#endif
+    addi            t2, t2, -1
+    add             a1, a1, a3
+    bgtz            t2, loop_scale2D_64to32
+
+    ret
+endfunc
+
+// void dequant_scaling_c(const int16_t* quantCoef, const int32_t* deQuantCoef, int16_t* coef, int num, int per, int shift)
+function PFX(dequant_scaling_v)
+    addi            a5, a5, 4
+    bge             a4, a5, dequant_scaling2
+    sub             a5, a5, a4
+loop_dequant_scaling1:
+    vsetvli         t1, a3, e32, m8, ta, ma
+    vle16.v         v0, (a0)
+    vle32.v         v16, (a1)
+    vsext.vf2       v8, v0
+    vmul.vv         v0, v8, v16
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vnclip.wx       v8, v0, a5
+    vse16.v         v8, (a2)
+    sub             a3, a3, t1
+    slli            t2, t1, 1
+    slli            t3, t1, 2
+    add             a0, a0, t2
+    add             a2, a2, t2
+    add             a1, a1, t3
+    bgtz            a3, loop_dequant_scaling1
+    ret
+    
+dequant_scaling2:
+    sub             a5, a4, a5
+loop_dequant_scaling2:
+    vsetvli         t1, a3, e32, m8, ta, ma
+    vle16.v         v0, (a0)
+    vle32.v         v16, (a1)
+    vsext.vf2       v8, v0
+    vmul.vv         v0, v8, v16
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vnclip.wi       v8, v0, 0
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vsext.vf2       v0, v8
+    vsll.vx         v0, v0, a5
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vnclip.wi       v8, v0, 0
+    vse16.v         v8, (a2)
+    sub             a3, a3, t1
+    slli            t2, t1, 1
+    slli            t3, t1, 2
+    add             a0, a0, t2
+    add             a2, a2, t2
+    add             a1, a1, t3
+    bgtz            a3, loop_dequant_scaling2
+    ret
+endfunc
+
+// void dequant_normal_c(const int16_t* quantCoef, int16_t* coef, int num, int scale, int shift)
+function PFX(dequant_normal_v)
+loop_dequant_normal:
+    vsetvli         t1, a2, e16, m4, ta, ma
+    slli            t2, t1, 1
+    vle16.v         v0, (a0)
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vsext.vf2       v16, v0
+    vmul.vx         v8, v16, a3
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vnclip.wx       v0, v8, a4
+    vse16.v         v0, (a1)
+    sub             a2, a2, t1
+    add             a0, a0, t2
+    add             a1, a1, t2
+    bgtz            a2, loop_dequant_normal
+    ret
+endfunc
+
+// void ssim_4x4x2_core(const pixel* pix1, intptr_t stride1, const pixel* pix2, intptr_t stride2, int sums24)
+function PFX(ssim_4x4x2_core_v)
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vmv.v.i         v4, 0
+    vmv.v.i         v6, 0
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v10, 0
+    vmv.v.i         v20, 0
+    li              t1, 4
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+#endif
+
+loop_ssim_4x4x2_core:
+#if HIGH_BIT_DEPTH
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vle16.v         v0, (a0)
+    vle16.v         v1, (a2)
+    vadd.vv         v4, v4, v0
+    vadd.vv         v6, v6, v1
+    vwmaccu.vv      v8, v0, v0
+    vwmaccu.vv      v8, v1, v1
+    vwmaccu.vv      v10, v0, v1
+#else
+    vsetivli        zero, 8, e8, m1, ta, ma
+    vle8.v          v0, (a0)
+    vle8.v          v1, (a2)
+    vwaddu.wv       v4, v4, v0
+    vwaddu.wv       v6, v6, v1
+    vwmulu.vv       v12, v0, v0
+    vwmulu.vv       v14, v1, v1
+    vwmulu.vv       v16, v0, v1
+    vsetivli        zero, 8, e16, m1, ta, ma
+    vwaddu.wv       v8, v8, v12
+    vwaddu.wv       v8, v8, v14
+    vwaddu.wv       v10, v10, v16
+#endif
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_ssim_4x4x2_core
+
+    vsetivli        zero, 4, e16, m1, ta, ma
+    vslidedown.vi   v0, v4, 4
+    vslidedown.vi   v1, v6, 4
+    vredsum.vs      v12, v4, v20
+    vredsum.vs      v13, v6, v20
+    vredsum.vs      v14, v0, v20
+    vredsum.vs      v15, v1, v20
+    vsetivli        zero, 4, e32, m1, ta, ma
+    vzext.vf2       v0, v12
+    vzext.vf2       v1, v13
+    vzext.vf2       v2, v14
+    vzext.vf2       v3, v15
+    vmv.x.s         t1, v0
+    vmv.x.s         t2, v1
+    vmv.x.s         t3, v2
+    vmv.x.s         t4, v3
+    sw              t1, (a4)
+    sw              t2, 4(a4)
+    sw              t3, 16(a4)
+    sw              t4, 20(a4)
+    vsetivli        zero, 8, e32, m2, ta, ma
+    vslidedown.vi   v12, v8, 4
+    vslidedown.vi   v14, v10, 4
+    vsetivli        zero, 4, e32, m1, ta, ma
+    vredsum.vs      v0, v8, v20
+    vredsum.vs      v1, v10, v20
+    vredsum.vs      v2, v12, v20
+    vredsum.vs      v3, v14, v20
+    vmv.x.s         t1, v0
+    vmv.x.s         t2, v1
+    vmv.x.s         t3, v2
+    vmv.x.s         t4, v3
+    sw              t1, 8(a4)
+    sw              t2, 12(a4)
+    sw              t3, 24(a4)
+    sw              t4, 28(a4)
+
+    ret
+endfunc
+
+// uint32_t quant_c(const int16_t* coef, const int32_t* quantCoeff, int32_t* deltaU, int16_t* qCoef, int qBits, int add, int numCoeff)
+function PFX(quant_v)
+    vsetvli         zero, a6, e32, m4, ta, ma
+    vmv.v.i         v24, 0
+    vmv.v.i         v28, 0
+    mv              t6, a6
+    addi            a7, a4, -8
+loop_quant:
+    vsetvli         t1, a6, e16, m2, ta, ma
+    vle16.v         v4, (a0)
+    VABS            v8, v4, v6
+    vsetvli         zero, t1, e32, m4, ta, ma
+    vzext.vf2       v12, v8
+    vsext.vf2       v16, v4
+    vmslt.vi        v0, v16, 0
+    vle32.v         v20, (a1)
+    vmul.vv         v12, v12, v20
+    vadd.vx         v8, v12, a5
+    vsra.vx         v8, v8, a4
+    vsll.vx         v4, v8, a4
+    vsub.vv         v4, v12, v4
+    vsra.vx         v4, v4, a7
+    vse32.v         v4, (a2)
+    vrsub.vi        v16, v8, 0
+    vmerge.vvm      v20, v8, v16, v0
+    vmsne.vi        v0, v8, 0
+    vmerge.vim      v12, v28, 1, v0
+    vsetvli         zero, t1, e32, m4, tu, ma
+    vadd.vv         v24, v24, v12
+    vsetvli         zero, t1, e16, m2, ta, ma
+    vnclip.wi       v4, v20, 0
+    vse16.v         v4, (a3)
+    sub             a6, a6, t1
+    slli            t2, t1, 1
+    slli            t3, t1, 2
+    add             a0, a0, t2
+    add             a3, a3, t2
+    add             a1, a1, t3
+    add             a2, a2, t3
+    bgtz            a6, loop_quant
+    
+    vsetvli         zero, t6, e32, m4, ta, ma
+    vredsum.vs      v16, v24, v28
+    vmv.x.s         a0, v16
+    ret
+endfunc
+
+// uint32_t nquant_c(const int16_t* coef, const int32_t* quantCoeff, int16_t* qCoef, int qBits, int add, int numCoeff)
+function PFX(nquant_v)
+    vsetvli         zero, a5, e32, m4, ta, ma
+    vmv.v.i         v24, 0
+    vmv.v.i         v28, 0
+    mv              t6, a5
+loop_nquant:
+    vsetvli         t1, a5, e16, m2, ta, ma
+    vle16.v         v4, (a0)
+    VABS            v8, v4, v6
+    vsetvli         zero, t1, e32, m4, ta, ma
+    vzext.vf2       v12, v8
+    vsext.vf2       v16, v4
+    vmslt.vi        v0, v16, 0
+    vle32.v         v20, (a1)
+    vmul.vv         v12, v12, v20
+    vadd.vx         v12, v12, a4
+    vsra.vx         v12, v12, a3
+    vrsub.vi        v16, v12, 0
+    vmerge.vvm      v20, v12, v16, v0
+    vmsne.vi        v0, v12, 0
+    vmerge.vim      v8, v28, 1, v0
+    vsetvli         zero, t1, e32, m4, tu, ma
+    vadd.vv         v24, v24, v8
+    vsetvli         zero, t1, e16, m2, ta, ma
+    vnclip.wi       v4, v20, 0
+    VABS            v12, v4, v8
+    vse16.v         v12, (a2)
+    sub             a5, a5, t1
+    slli            t2, t1, 1
+    slli            t3, t1, 2
+    add             a0, a0, t2
+    add             a2, a2, t2
+    add             a1, a1, t3
+    bgtz            a5, loop_nquant
+    
+    vsetvli         zero, t6, e32, m4, ta, ma
+    vredsum.vs      v16, v24, v28
+    vmv.x.s         a0, v16
+    ret
+endfunc
+
+// void ssimDist_c(const pixel* fenc, uint32_t fStride, const pixel* recon, intptr_t rstride, uint64_t *ssBlock, int shift, uint64_t *ac_k)
+.macro SSIM_DIST size, lmul, lmul2, lmul4
+function PFX(ssimDist_\size\()_v)
+    li              t1, \size
+    li              t2, \size
+    vsetvli         zero, t2, e32, m\lmul4, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+    vmv.v.x         v28, a5
+    vmv1r.v         v31, v8
+loop_ssimDist\size\():
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+    vle8.v          v0, (a0)
+    vle8.v          v2, (a2)
+    vwsubu.vv       v4, v0, v2
+    vsrl.vv         v0, v0, v28
+    vsetvli         zero, t2, e16, m\lmul2, ta, ma
+    vwmacc.vv       v8, v4, v4
+    vzext.vf2       v4, v0
+    vwmacc.vv       v16, v4, v4
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_ssimDist\size
+    vsetvli         zero, t2, e32, m\lmul4, ta, ma
+    vredsum.vs      v0, v8, v31
+    vredsum.vs      v1, v16, v31
+    vmv.x.s         t1, v0
+    vmv.x.s         t2, v1
+    sd              t1, (a4)
+    sd              t2, (a6)
+    ret
+endfunc
+.endm
+
+.macro SSIM_DIST2 size, lmul, lmul2, lmul4
+function PFX(ssimDist_\size\()_v)
+    li              t1, \size
+    li              t2, \size
+    vsetvli         zero, t2, e32, m\lmul4, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+    vmv.v.x         v30, a5
+    vmv1r.v         v31, v8
+loop_ssimDist\size\():
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+    add             t3, a0, a1
+    add             t4, a2, a3
+    vle8.v          v0, (a0)
+    vle8.v          v4, (t3)
+    vle8.v          v2, (a2)
+    vle8.v          v6, (t4)
+    add             a0, t3, a1
+    add             a2, t4, a3
+    vwsubu.vv       v12, v0, v2
+    vwsubu.vv       v20, v4, v6
+    vsrl.vv         v0, v0, v30
+    vsrl.vv         v2, v4, v30
+    vsetvli         zero, t2, e16, m\lmul2, ta, ma
+    vwmacc.vv       v8, v12, v12
+    vwmacc.vv       v8, v20, v20
+    vzext.vf2       v4, v0
+    vzext.vf2       v24, v2
+    vwmacc.vv       v16, v4, v4
+    vwmacc.vv       v16, v24, v24
+    addi            t1, t1, -2
+    bgtz            t1, loop_ssimDist\size
+    vsetvli         zero, t2, e32, m\lmul4, ta, ma
+    vredsum.vs      v0, v8, v31
+    vredsum.vs      v1, v16, v31
+    vmv.x.s         t1, v0
+    vmv.x.s         t2, v1
+    sd              t1, (a4)
+    sd              t2, (a6)
+    ret
+endfunc
+.endm
+
+SSIM_DIST2 4,  1, 1, 1
+SSIM_DIST2 8,  1, 1, 2
+SSIM_DIST2 16, 1, 2, 4
+SSIM_DIST  32, 2, 4, 8
+
+// void ssimDist_c(const pixel* fenc, uint32_t fStride, const pixel* recon, intptr_t rstride, uint64_t *ssBlock, int shift, uint64_t *ac_k)
+function PFX(ssimDist_64_v)
+    li              t1, 64
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vmv.v.i         v8, 0
+    vmv.v.i         v16, 0
+    vmv1r.v         v31, v8
+loop_ssimDist642:
+    li              t2, 64
+    mv              t4, a0
+    mv              t5, a2
+loop_ssimDist641:
+    vsetvli         t3, t2, e8, m2, ta, ma
+    vle8.v          v0, (t4)
+    vle8.v          v2, (t5)
+    vwsubu.vv       v4, v0, v2
+    vsrl.vx         v0, v0, a5
+    vsetvli         zero, t3, e16, m4, ta, ma
+    vwmacc.vv       v8, v4, v4
+    vzext.vf2       v4, v0
+    vwmacc.vv       v16, v4, v4
+    add             t4, t4, t3
+    add             t5, t5, t3
+    sub             t2, t2, t3
+    bgtz            t2, loop_ssimDist641
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_ssimDist642
+    vsetvli         zero, t3, e32, m8, ta, ma
+    vredsum.vs      v0, v8, v31
+    vredsum.vs      v1, v16, v31
+    vmv.x.s         t1, v0
+    vmv.x.s         t2, v1
+    sd              t1, (a4)
+    sd              t2, (a6)
+    ret
+endfunc
+
+// void normFact_c(const pixel* src, uint32_t blockSize, int shift, uint64_t *z_k)
+function PFX(normFact_v)
+    mul             a1, a1, a1
+    vsetvli         zero, a1, e32, m8, ta, ma
+    vmv.v.i         v24, 0
+    vmv1r.v         v20, v24
+loop_normFact:
+    vsetvli         t1, a1, e8, m2, ta, ma
+    vle8.v          v0, (a0)
+    vsrl.vx         v0, v0, a2
+    vwmulu.vv       v4, v0, v0
+    vsetvli         zero, t1, e16, m4, ta, ma
+    vwaddu.wv       v24, v24, v4
+    sub             a1, a1, t1
+    add             a0, a0, t1
+    bgtz            a1, loop_normFact
+    vsetvli         zero, t1, e32, m8, ta, ma
+    vredsum.vs      v8, v24, v20
+    vmv.x.s         t2, v8
+    sd              t2, (a3)
+    ret
+endfunc
+
+// int scanPosLast_c(const uint16_t *scan, const coeff_t *coeff, uint16_t *coeffSign,
+//    uint16_t *coeffFlag, uint8_t *coeffNum, int numSig, const uint16_t*, const int)
+function PFX(scanPosLast_v)
+    li              t1, 64
+    li              t6, 0
+    vsetvli         zero, t1, e16, m8, ta, ma
+    vmv.v.i         v16, 0
+    vse16.v         v16, (a2)
+    vse16.v         v16, (a3)
+    vse8.v          v16, (a4)
+    li              t3, 0x8000
+loop_scanPosLast:
+    vsetivli        zero, 16, e16, m2, ta, ma
+    vle16.v         v2, (a0)
+    vsll.vi         v2, v2, 1
+    vluxei16.v      v4, (a1), v2
+    vmsne.vi        v0, v4, 0
+    viota.m         v30, v0
+    vcpop.m         t2, v0
+    vmerge.vim      v12, v16, 1, v0
+    vid.v           v14
+    vsrl.vi         v4, v4, 15
+    vsll.vv         v4, v4, v30
+    bge             t2, a5, tail_scanPosLast
+    vrsub.vi        v14, v14, 15
+    vsll.vv         v12, v12, v14
+    vredor.vs       v2, v12, v16
+    vredsum.vs      v10, v4, v16
+    vmv.x.s         t5, v2
+    vmv.x.s         t4, v10
+    sh              t4, (a2)
+    sb              t2, (a4)
+    sh              t5, (a3)
+    sub             a5, a5, t2
+    addi            a0, a0, 32
+    addi            a2, a2, 2
+    addi            a3, a3, 2
+    addi            a4, a4, 1
+    addi            t6, t6, 16
+    bgtz            a5, loop_scanPosLast
+tail_scanPosLast:
+    li              t0, 16
+    vmv.x.s         t5, v0
+    and             t4, t3, t5
+    sub             t1, t2, a5
+    add             t1, t1, t4
+    beq             t1, t3, finish_scanPosLast
+    vmseq.vx        v28, v30, a5
+    vmsbf.m         v0, v28
+    vcpop.m         t0, v0
+finish_scanPosLast:
+    vsetvli         zero, t0, e16, m2, ta, ma
+    addi            t1, t0, -1
+    vrsub.vx        v14, v14, t1
+    vsll.vv         v12, v12, v14
+    vredor.vs       v2, v12, v16
+    vredsum.vs      v10, v4, v16
+    vmv.x.s         t4, v10
+    vmv.x.s         t5, v2
+    sh              t4, (a2)
+    sh              t5, (a3)
+    sb              a5, (a4)
+    add             a0, t6, t1
+    ret
+endfunc
+
+//void weight_pp_c(const pixel* src, pixel* dst, intptr_t stride, int width, int height, int w0, int round, int shift, int offset)
+function PFX(weight_pp_v)
+    ld              t0, (sp)
+    li              t6, (1 << BIT_DEPTH) - 1
+    addi            a7, a7, BIT_DEPTH - 14
+    addi            t1, a7, -1
+    li              t2, 1
+    sll             a6, t2, t1
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+    vsetvli         zero, a3, e16, m1, ta, ma
+    vmv.v.x         v1, a5
+    vsetvli         zero, a3, e32, m2, ta, ma
+#else
+    sll             t3, t2, a7
+    addi            t3, t3, -1
+    and             t3, t3, a5
+    beqz            t3, fast_weight_pp
+    vsetvli         zero, a3, e8, m1, ta, ma
+    vmv.v.x         v1, a5
+    vsetvli         zero, a3, e32, m4, ta, ma
+#endif
+    vmv.v.i         v12, 0
+    vmv.v.x         v16, t6
+    vmv.v.x         v20, a6
+    vmv.v.x         v24, a7
+    vmv.v.x         v28, t0
+loop_weight_pp_h:
+    mv              t1, a0
+    mv              t2, a1
+    mv              t3, a3
+loop_weight_pp_w:
+#if HIGH_BIT_DEPTH
+    vsetvli         t4, t3, e16, m1, ta, ma
+    vle16.v         v0, (t1)
+    vwmul.vv        v4, v0, v1
+    vsetvli         zero, t4, e32, m2, ta, ma
+#else
+    vsetvli         t4, t3, e8, m1, ta, ma
+    vle8.v          v0, (t1)
+    vwmulu.vv       v2, v0, v1
+    vsetvli         zero, t4, e32, m4, ta, ma
+    vzext.vf2       v4, v2
+#endif
+    vadd.vv         v4, v4, v20
+    vsra.vv         v4, v4, v24
+    vadd.vv         v4, v4, v28
+    vmax.vv         v4, v4, v12
+    vmin.vv         v4, v4, v16
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, t4, e16, m1, ta, ma
+    vnsrl.wi        v8, v4, 0
+    vse16.v         v8, (t2)
+#else
+    vsetvli         zero, t4, e16, m2, ta, ma
+    vnsrl.wi        v8, v4, 0
+    vsetvli         zero, t4, e8, m1, ta, ma
+    vnsrl.wi        v4, v8, 0
+    vse8.v          v4, (t2)
+#endif
+    sub             t3, t3, t4
+#if HIGH_BIT_DEPTH
+    slli            t4, t4, 1
+#endif
+    add             t1, t1, t4
+    add             t2, t2, t4
+    bgtz            t3, loop_weight_pp_w
+    addi            a4, a4, -1
+    add             a0, a0, a2
+    add             a1, a1, a2
+    bgtz            a4, loop_weight_pp_h
+    ret
+fast_weight_pp:
+    add             a5, a5, a6
+    sra             a5, a5, a7
+    vsetvli         zero, a3, e8, m1, ta, ma
+    vmv.v.x         v1, a5
+    vsetvli         zero, a3, e16, m2, ta, ma
+    vmv.v.i         v12, 0
+    vmv.v.x         v14, t6
+    vmv.v.x         v16, t0
+loop_weight_pp_fh:
+    mv              t1, a0
+    mv              t2, a1
+    mv              t3, a3
+loop_weight_pp_fw:
+    vsetvli         t4, t3, e8, m1, ta, ma
+    vle8.v          v0, (t1)
+    vwmulu.vv       v2, v0, v1
+    vsetvli         zero, t4, e16, m2, ta, ma
+    vadd.vv         v2, v2, v16
+    vmax.vv         v2, v2, v12
+    vmin.vv         v2, v2, v14
+    vsetvli         zero, t4, e8, m1, ta, ma
+    vnsrl.wi        v4, v2, 0
+    vse8.v          v4, (t2)
+    sub             t3, t3, t4
+    add             t1, t1, t4
+    add             t2, t2, t4
+    bgtz            t3, loop_weight_pp_fw
+    addi            a4, a4, -1
+    add             a0, a0, a2
+    add             a1, a1, a2
+    bgtz            a4, loop_weight_pp_fh
+    ret
+endfunc
+
+//void weight_sp_c(const int16_t* src, pixel* dst, intptr_t srcStride, intptr_t dstStride, int width, int height, int w0, int round, int shift, int offset)
+function PFX(weight_sp_v)
+    ld              t0, (sp)
+    ld              t1, 8(sp)
+    li              t6, (1 << BIT_DEPTH) - 1
+    li              t2, 1 << 13
+    mul             t2, t2, a6
+    add             a7, a7, t2
+    slli            a2, a2, 1
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+#endif
+    vsetvli         zero, a3, e16, m1, ta, ma
+    vmv.v.x         v1, a6
+    vsetvli         zero, a3, e32, m2, ta, ma
+    vmv.v.i         v12, 0
+    vmv.v.x         v16, t6
+    vmv.v.x         v20, a7
+    vmv.v.x         v24, t0
+    vmv.v.x         v28, t1
+loop_weight_sp_h:
+    mv              t1, a0
+    mv              t2, a1
+    mv              t3, a4
+loop_weight_sp_w:
+    vsetvli         t4, t3, e16, m1, ta, ma
+    vle16.v         v0, (t1)
+    vwmul.vv        v4, v0, v1
+    vsetvli         zero, t4, e32, m2, ta, ma
+    vadd.vv         v4, v4, v20
+    vsra.vv         v4, v4, v24
+    vadd.vv         v4, v4, v28
+    vmax.vv         v4, v4, v12
+    vmin.vv         v4, v4, v16
+    vsetvli         zero, t4, e16, m1, ta, ma
+    vnsrl.wi        v8, v4, 0
+#if HIGH_BIT_DEPTH
+    vse16.v         v8, (t2)
+#else
+    vsetvli         zero, t4, e8, m1, ta, ma
+    vnsrl.wi        v4, v8, 0
+    vse8.v          v4, (t2)
+#endif
+    sub             t3, t3, t4
+    slli            t5, t4, 1
+#if HIGH_BIT_DEPTH
+    add             t2, t2, t5
+#else
+    add             t2, t2, t4
+#endif
+    add             t1, t1, t5
+    bgtz            t3, loop_weight_sp_w
+    addi            a5, a5, -1
+    add             a0, a0, a2
+    add             a1, a1, a3
+    bgtz            a5, loop_weight_sp_h
+    ret
+endfunc
+
+//void addAvg(const int16_t* src0, const int16_t* src1, pixel* dst, intptr_t src0Stride, intptr_t src1Stride, intptr_t dstStride)
+.macro ADDAVG_X_Y bx, by, lmul1, lmul2, lmul3
+function PFX(addAvg_\bx\()x\by\()_v)
+#if HIGH_BIT_DEPTH
+    slli            a5, a5, 1
+#endif
+    slli            a3, a3, 1
+    slli            a4, a4, 1
+    li              a6, \by
+    li              a7, (1 << (14 - BIT_DEPTH)) + 2 * (1 << 13)
+    li              t6, (1 << BIT_DEPTH) - 1
+    li              t1, \bx
+    vsetvli         zero, t1, e32, m\lmul3, ta, ma
+    vmv.v.i         v28, 0
+    vmv.v.x         v24, t6
+    vmv.v.x         v20, a7
+loop_addAvg_\bx\()_\by\()_y:
+    li              t1, \bx
+    mv              t2, a0
+    mv              t3, a1
+    mv              t4, a2
+loop_addAvg_\bx\()_\by\()_x:
+    vsetvli         t5, t1, e16, m\lmul2, ta, ma
+    vle16.v         v0, (t2)
+    vle16.v         v8, (t3)
+    vwadd.vv        v4, v0, v8
+    vsetvli         zero, t5, e32, m\lmul3, ta, ma
+    vadd.vv         v4, v4, v20
+    vsra.vi         v4, v4, (15 - BIT_DEPTH)
+    vmax.vv         v4, v4, v28
+    vmin.vv         v4, v4, v24
+    vsetvli         zero, t5, e16, m\lmul2, ta, ma
+    vnsrl.wi        v8, v4, 0
+#if HIGH_BIT_DEPTH
+    vse16.v         v8, (t4)
+#else
+    vsetvli         zero, t5, e8, m\lmul1, ta, ma
+    vnsrl.wi        v4, v8, 0
+    vse8.v          v4, (t4)
+#endif
+    slli            t6, t5, 1
+    sub             t1, t1, t5
+    add             t2, t2, t6
+    add             t3, t3, t6
+#if HIGH_BIT_DEPTH
+    add             t4, t4, t6
+#else
+    add             t4, t4, t5
+#endif
+    bgtz            t1, loop_addAvg_\bx\()_\by\()_x
+    add             a0, a0, a3
+    add             a1, a1, a4
+    add             a2, a2, a5
+    addi            a6, a6, -1
+    bgtz            a6, loop_addAvg_\bx\()_\by\()_y
+    ret
+endfunc
+.endm
+
+.macro ADDAVG_X_Y2 bx, by, lmul1, lmul2, lmul3
+function PFX(addAvg_\bx\()x\by\()_v)
+#if HIGH_BIT_DEPTH
+    slli            a5, a5, 1
+#endif
+    slli            a3, a3, 1
+    slli            a4, a4, 1
+    li              a6, \by
+    li              a7, (1 << (14 - BIT_DEPTH)) + 2 * (1 << 13)
+    li              t6, (1 << BIT_DEPTH) - 1
+    li              t1, \bx
+    vsetvli         zero, t1, e32, m\lmul3, ta, ma
+    vmv.v.i         v28, 0
+    vmv.v.x         v24, t6
+    vmv.v.x         v20, a7
+loop_addAvg_\bx\()_\by\()_y:
+    vsetvli         t5, t1, e16, m\lmul2, ta, ma
+    add             t2, a0, a3
+    add             t3, a1, a4
+    vle16.v         v0, (a0)
+    vle16.v         v4, (t2)
+    vle16.v         v8, (a1)
+    vle16.v         v12, (t3)
+    add             a0, t2, a3
+    add             a1, t3, a4
+    vwadd.vv        v16, v0, v8
+    vwadd.vv        v8, v4, v12
+    vsetvli         zero, t5, e32, m\lmul3, ta, ma
+    vadd.vv         v16, v16, v20
+    vadd.vv         v8, v8, v20
+    vsra.vi         v16, v16, (15 - BIT_DEPTH)
+    vsra.vi         v8, v8, (15 - BIT_DEPTH)
+    vmax.vv         v16, v16, v28
+    vmax.vv         v8, v8, v28
+    vmin.vv         v16, v16, v24
+    vmin.vv         v8, v8, v24
+    vsetvli         zero, t5, e16, m\lmul2, ta, ma
+    vnsrl.wi        v0, v16, 0
+    vnsrl.wi        v4, v8, 0
+    add             t3, a2, a5
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (a2)
+    vse16.v         v4, (t3)
+#else
+    vsetvli         zero, t5, e8, m\lmul1, ta, ma
+    vnsrl.wi        v8, v0, 0
+    vnsrl.wi        v12, v4, 0
+    vse8.v          v8, (a2)
+    vse8.v          v12, (t3)
+#endif
+    add             a2, t3, a5
+    addi            a6, a6, -2
+    bgtz            a6, loop_addAvg_\bx\()_\by\()_y
+    ret
+endfunc
+.endm
+
+ADDAVG_X_Y2  4, 2, 1, 1, 1
+ADDAVG_X_Y2  4, 4, 1, 1, 1
+ADDAVG_X_Y2  4, 8, 1, 1, 1
+ADDAVG_X_Y2  4, 16, 1, 1, 1
+ADDAVG_X_Y2  4, 32, 1, 1, 1
+ADDAVG_X_Y2  6, 8, 1, 1, 2
+ADDAVG_X_Y2  6, 16, 1, 1, 2
+ADDAVG_X_Y2  8, 2, 1, 1, 2
+ADDAVG_X_Y2  8, 4, 1, 1, 2
+ADDAVG_X_Y2  8, 6, 1, 1, 2
+ADDAVG_X_Y2  8, 8, 1, 1, 2
+ADDAVG_X_Y2  8, 12, 1, 1, 2
+ADDAVG_X_Y2  8, 16, 1, 1, 2
+ADDAVG_X_Y2  8, 32, 1, 1, 2
+ADDAVG_X_Y2  8, 64, 1, 1, 2
+ADDAVG_X_Y2  12, 16, 1, 2, 4
+ADDAVG_X_Y2  12, 32, 1, 2, 4
+ADDAVG_X_Y2  16, 4, 1, 2, 4
+ADDAVG_X_Y2  16, 8, 1, 2, 4
+ADDAVG_X_Y2  16, 12, 1, 2, 4
+ADDAVG_X_Y2  16, 16, 1, 2, 4
+ADDAVG_X_Y2  16, 24, 1, 2, 4
+ADDAVG_X_Y2  16, 32, 1, 2, 4
+ADDAVG_X_Y2  16, 64, 1, 2, 4
+ADDAVG_X_Y  24, 32, 1, 2, 4
+ADDAVG_X_Y  24, 64, 1, 2, 4
+ADDAVG_X_Y  32, 8, 1, 2, 4
+ADDAVG_X_Y  32, 12, 1, 2, 4
+ADDAVG_X_Y  32, 16, 1, 2, 4
+ADDAVG_X_Y  32, 24, 1, 2, 4
+ADDAVG_X_Y  32, 32, 1, 2, 4
+ADDAVG_X_Y  32, 48, 1, 2, 4
+ADDAVG_X_Y  32, 64, 1, 2, 4
+ADDAVG_X_Y  48, 64, 1, 2, 4
+ADDAVG_X_Y  64, 16, 1, 2, 4
+ADDAVG_X_Y  64, 32, 1, 2, 4
+ADDAVG_X_Y  64, 48, 1, 2, 4
+ADDAVG_X_Y  64, 64, 1, 2, 4
+
+//void blockfill_s_c(int16_t* dst, intptr_t dstride, int16_t val)
+.macro BLOCKFILL_S size, lmul
+function PFX(blockfill_s_\size\()_v)
+    slli            a1, a1, 1
+    li              t1, \size
+    slli            a4, a1, 1
+    add             a5, a4, a1
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a2
+loop_blockfill_s_\size:
+    add             t3, a0, a1
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vse16.v         v0, (a0)
+    vse16.v         v0, (t3)
+    vse16.v         v0, (t4)
+    vse16.v         v0, (t5)
+    add             a0, t5, a1
+    addi            t1, t1, -4
+    bgtz            t1, loop_blockfill_s_\size
+    ret
+endfunc
+.endm
+
+BLOCKFILL_S 4, 1
+BLOCKFILL_S 8, 1
+BLOCKFILL_S 16, 2
+BLOCKFILL_S 32, 4
+BLOCKFILL_S 64, 8
+
+//void cpy2Dto1D_shl(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
+.macro CPY2DTO1D_SHL size, lmul
+function PFX(cpy2Dto1D_shl_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+cpy2Dto1D_shl_\size:
+    vle16.v         v8, (a1)
+    vsll.vv         v8, v8, v0
+    vse16.v         v8, (a0)
+    addi            a0, a0, 2 * \size
+    add             a1, a1, a2
+    addi            t1, t1, -1
+    bgtz            t1, cpy2Dto1D_shl_\size
+    ret
+endfunc
+.endm
+
+.macro CPY2DTO1D_SHL2 size, lmul
+function PFX(cpy2Dto1D_shl_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    slli            a4, a2, 1
+    add             a5, a4, a2
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+cpy2Dto1D_shl_\size:
+    add             t3, a1, a2
+    add             t4, a1, a4
+    add             t5, a1, a5
+    vle16.v         v4, (a1)
+    vle16.v         v8, (t3)
+    vle16.v         v12, (t4)
+    vle16.v         v16, (t5)
+    add             a1, t5, a2
+    vsll.vv         v4, v4, v0
+    vsll.vv         v8, v8, v0
+    vsll.vv         v12, v12, v0
+    vsll.vv         v16, v16, v0
+    addi            t3, a0, 2 * \size
+    addi            t4, a0, 4 * \size
+    addi            t5, a0, 6 * \size
+    vse16.v         v4, (a0)
+    vse16.v         v8, (t3)
+    vse16.v         v12, (t4)
+    vse16.v         v16, (t5)
+    addi            a0, t5, 2 * \size
+    addi            t1, t1, -4
+    bgtz            t1, cpy2Dto1D_shl_\size
+    ret
+endfunc
+.endm
+
+CPY2DTO1D_SHL2 4, 1
+CPY2DTO1D_SHL2 8, 1
+CPY2DTO1D_SHL2 16, 2
+CPY2DTO1D_SHL2 32, 4
+CPY2DTO1D_SHL  64, 8
+
+//void cpy2Dto1D_shr(int16_t* dst, const int16_t* src, intptr_t srcStride, int shift)
+.macro CPY2DTO1D_SHR size, lmul
+function PFX(cpy2Dto1D_shr_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    addi            t2, a3, -1
+    li              t3, 1
+    sll             t3, t3, t2
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+    vmv.v.x         v24, t3
+cpy2Dto1D_shr_\size:
+    vle16.v         v8, (a1)
+    vadd.vv         v8, v8, v24
+    vsra.vv         v8, v8, v0
+    vse16.v         v8, (a0)
+    addi            a0, a0, 2 * \size
+    add             a1, a1, a2
+    addi            t1, t1, -1
+    bgtz            t1, cpy2Dto1D_shr_\size
+    ret
+endfunc
+.endm
+
+.macro CPY2DTO1D_SHR2 size, lmul
+function PFX(cpy2Dto1D_shr_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    addi            t2, a3, -1
+    li              t3, 1
+    sll             t3, t3, t2
+    slli            a4, a2, 1
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+    vmv.v.x         v24, t3
+    add             a5, a4, a2
+cpy2Dto1D_shr_\size:
+    add             t3, a1, a2
+    add             t4, a1, a4
+    add             t5, a1, a5
+    vle16.v         v4, (a1)
+    vle16.v         v8, (t3)
+    vle16.v         v12, (t4)
+    vle16.v         v16, (t5)
+    add             a1, t5, a2
+    vadd.vv         v4, v4, v24
+    vadd.vv         v8, v8, v24
+    vadd.vv         v12, v12, v24
+    vadd.vv         v16, v16, v24
+    vsra.vv         v4, v4, v0
+    vsra.vv         v8, v8, v0
+    vsra.vv         v12, v12, v0
+    vsra.vv         v16, v16, v0
+    addi            t3, a0, 2 * \size
+    addi            t4, a0, 4 * \size
+    addi            t5, a0, 6 * \size
+    vse16.v         v4, (a0)
+    vse16.v         v8, (t3)
+    vse16.v         v12, (t4)
+    vse16.v         v16, (t5)
+    addi            a0, t5, 2 * \size
+    addi            t1, t1, -4
+    bgtz            t1, cpy2Dto1D_shr_\size
+    ret
+endfunc
+.endm
+
+CPY2DTO1D_SHR2 4, 1
+CPY2DTO1D_SHR2 8, 1
+CPY2DTO1D_SHR2 16, 2
+CPY2DTO1D_SHR2 32, 4
+CPY2DTO1D_SHR  64, 8
+
+//void cpy1Dto2D_shl(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
+.macro CPY1DTO2D_SHL size, lmul
+function PFX(cpy1Dto2D_shl_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+cpy1Dto2D_shl_\size:
+    vle16.v         v8, (a1)
+    vsll.vv         v8, v8, v0
+    vse16.v         v8, (a0)
+    addi            a1, a1, 2 * \size
+    add             a0, a0, a2
+    addi            t1, t1, -1
+    bgtz            t1, cpy1Dto2D_shl_\size
+    ret
+endfunc
+.endm
+
+.macro CPY1DTO2D_SHL2 size, lmul
+function PFX(cpy1Dto2D_shl_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    slli            a4, a2, 1
+    add             a5, a4, a2
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+cpy1Dto2D_shl_\size:
+    addi            t3, a1, 2 * \size
+    addi            t4, a1, 4 * \size
+    addi            t5, a1, 6 * \size
+    vle16.v         v4, (a1)
+    vle16.v         v8, (t3)
+    vle16.v         v12, (t4)
+    vle16.v         v16, (t5)
+    addi            a1, t5, 2 * \size
+    vsll.vv         v4, v4, v0
+    vsll.vv         v8, v8, v0
+    vsll.vv         v12, v12, v0
+    vsll.vv         v16, v16, v0
+    add             t3, a0, a2
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vse16.v         v4, (a0)
+    vse16.v         v8, (t3)
+    vse16.v         v12, (t4)
+    vse16.v         v16, (t5)
+    add             a0, t5, a2
+    addi            t1, t1, -4
+    bgtz            t1, cpy1Dto2D_shl_\size
+    ret
+endfunc
+.endm
+
+CPY1DTO2D_SHL2 4, 1
+CPY1DTO2D_SHL2 8, 1
+CPY1DTO2D_SHL2 16, 2
+CPY1DTO2D_SHL2 32, 4
+CPY1DTO2D_SHL  64, 8
+
+//void cpy1Dto2D_shr(int16_t* dst, const int16_t* src, intptr_t dstStride, int shift)
+.macro CPY1DTO2D_SHR size, lmul
+function PFX(cpy1Dto2D_shr_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    addi            t2, a3, -1
+    li              t3, 1
+    sll             t3, t3, t2
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+    vmv.v.x         v24, t3
+cpy1Dto2D_shr_\size:
+    vle16.v         v8, (a1)
+    vadd.vv         v8, v8, v24
+    vsra.vv         v8, v8, v0
+    vse16.v         v8, (a0)
+    addi            a1, a1, 2 * \size
+    add             a0, a0, a2
+    addi            t1, t1, -1
+    bgtz            t1, cpy1Dto2D_shr_\size
+    ret
+endfunc
+.endm
+
+.macro CPY1DTO2D_SHR2 size, lmul
+function PFX(cpy1Dto2D_shr_\size\()_v)
+    slli            a2, a2, 1
+    li              t1, \size
+    addi            t2, a3, -1
+    li              t3, 1
+    sll             t3, t3, t2
+    slli            a4, a2, 1
+    vsetvli         zero, t1, e16, m\lmul, ta, ma
+    vmv.v.x         v0, a3
+    vmv.v.x         v24, t3
+    add             a5, a4, a2
+cpy1Dto2D_shr_\size:
+    addi            t3, a1, 2 * \size
+    addi            t4, a1, 4 * \size
+    addi            t5, a1, 6 * \size
+    vle16.v         v4, (a1)
+    vle16.v         v8, (t3)
+    vle16.v         v12, (t4)
+    vle16.v         v16, (t5)
+    addi            a1, t5, 2 * \size
+    vadd.vv         v4, v4, v24
+    vadd.vv         v8, v8, v24
+    vadd.vv         v12, v12, v24
+    vadd.vv         v16, v16, v24
+    vsra.vv         v4, v4, v0
+    vsra.vv         v8, v8, v0
+    vsra.vv         v12, v12, v0
+    vsra.vv         v16, v16, v0
+    add             t3, a0, a2
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vse16.v         v4, (a0)
+    vse16.v         v8, (t3)
+    vse16.v         v12, (t4)
+    vse16.v         v16, (t5)
+    add             a0, t5, a2
+    addi            t1, t1, -4
+    bgtz            t1, cpy1Dto2D_shr_\size
+    ret
+endfunc
+.endm
+
+CPY1DTO2D_SHR2 4, 1
+CPY1DTO2D_SHR2 8, 1
+CPY1DTO2D_SHR2 16, 2
+CPY1DTO2D_SHR2 32, 4
+CPY1DTO2D_SHR  64, 8
+
+//void blockcopy_pp_c(pixel* a, intptr_t stridea, const pixel* b, intptr_t strideb)
+.macro BLOCKCOPY_PP bx, by, lmul
+function PFX(blockcopy_pp_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+#else
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+#endif
+loop_blockcopy_pp_\bx\()x\by:
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vse16.v         v0, (a0)
+#else
+    vle8.v          v0, (a2)
+    vse8.v          v0, (a0)
+#endif
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_blockcopy_pp_\bx\()x\by
+    ret
+endfunc
+.endm
+
+.macro BLOCKCOPY_PP2 bx, by, lmul
+function PFX(blockcopy_pp_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+#else
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+#endif
+    slli            a4, a1, 1
+    slli            a6, a3, 1
+    add             a5, a4, a1
+    add             a7, a6, a3
+loop_blockcopy_pp_\bx\()x\by:
+    add             t3, a2, a3
+    add             t4, a2, a6
+    add             t5, a2, a7
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+    vle16.v         v8, (t4)
+    vle16.v         v12, (t5)
+#else
+    vle8.v          v0, (a2)
+    vle8.v          v4, (t3)
+    vle8.v          v8, (t4)
+    vle8.v          v12, (t5)
+#endif
+    add             a2, t5, a3
+    add             t3, a0, a1
+    add             t4, a0, a4
+    add             t5, a0, a5
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (a0)
+    vse16.v         v4, (t3)
+    vse16.v         v8, (t4)
+    vse16.v         v12, (t5)
+#else
+    vse8.v          v0, (a0)
+    vse8.v          v4, (t3)
+    vse8.v          v8, (t4)
+    vse8.v          v12, (t5)
+#endif
+    add             a0, t5, a1
+    addi            t1, t1, -4
+    bgtz            t1, loop_blockcopy_pp_\bx\()x\by
+    ret
+endfunc
+.endm
+
+.macro BLOCKCOPY_PP3 bx, by, lmul
+function PFX(blockcopy_pp_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+#else
+    vsetvli         zero, t2, e8, m\lmul, ta, ma
+#endif
+loop_blockcopy_pp_\bx\()x\by:
+    add             t3, a2, a3
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+#else
+    vle8.v          v0, (a2)
+    vle8.v          v4, (t3)
+#endif
+    add             a2, t3, a3
+    add             t4, a0, a1
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (a0)
+    vse16.v         v4, (t4)
+#else
+    vse8.v          v0, (a0)
+    vse8.v          v4, (t4)
+#endif
+    add             a0, t4, a1
+    addi            t1, t1, -2
+    bgtz            t1, loop_blockcopy_pp_\bx\()x\by
+    ret
+endfunc
+.endm
+
+BLOCKCOPY_PP3  4, 2, 1
+BLOCKCOPY_PP2  4, 4, 1
+BLOCKCOPY_PP2  4, 8, 1
+BLOCKCOPY_PP2  4, 16, 1
+BLOCKCOPY_PP2  4, 32, 1
+BLOCKCOPY_PP2  6, 8, 1
+BLOCKCOPY_PP2  6, 16, 1
+BLOCKCOPY_PP3  8, 2, 1
+BLOCKCOPY_PP2  8, 4, 1
+BLOCKCOPY_PP3  8, 6, 1
+BLOCKCOPY_PP2  8, 8, 1
+BLOCKCOPY_PP2  8, 12, 1
+BLOCKCOPY_PP2  8, 16, 1
+BLOCKCOPY_PP2  8, 32, 1
+BLOCKCOPY_PP2  8, 64, 1
+#if HIGH_BIT_DEPTH
+BLOCKCOPY_PP2  12, 16, 2
+BLOCKCOPY_PP2  12, 32, 2
+BLOCKCOPY_PP2  16, 4, 2
+BLOCKCOPY_PP2  16, 8, 2
+BLOCKCOPY_PP2  16, 12, 2
+BLOCKCOPY_PP2  16, 16, 2
+BLOCKCOPY_PP2  16, 24, 2
+BLOCKCOPY_PP2  16, 32, 2
+BLOCKCOPY_PP2  16, 64, 2
+BLOCKCOPY_PP2  24, 32, 4
+BLOCKCOPY_PP2  24, 64, 4
+BLOCKCOPY_PP2  32, 8, 4
+BLOCKCOPY_PP2  32, 12, 4
+BLOCKCOPY_PP2  32, 16, 4
+BLOCKCOPY_PP2  32, 24, 4
+BLOCKCOPY_PP2  32, 32, 4
+BLOCKCOPY_PP2  32, 48, 4
+BLOCKCOPY_PP2  32, 64, 4
+BLOCKCOPY_PP   48, 64, 8
+BLOCKCOPY_PP   64, 16, 8
+BLOCKCOPY_PP   64, 32, 8
+BLOCKCOPY_PP   64, 48, 8
+BLOCKCOPY_PP   64, 64, 8
+#else
+BLOCKCOPY_PP2  12, 16, 1
+BLOCKCOPY_PP2  12, 32, 1
+BLOCKCOPY_PP2  16, 4, 1
+BLOCKCOPY_PP2  16, 8, 1
+BLOCKCOPY_PP2  16, 12, 1
+BLOCKCOPY_PP2  16, 16, 1
+BLOCKCOPY_PP2  16, 24, 1
+BLOCKCOPY_PP2  16, 32, 1
+BLOCKCOPY_PP2  16, 64, 1
+BLOCKCOPY_PP2  24, 32, 2
+BLOCKCOPY_PP2  24, 64, 2
+BLOCKCOPY_PP2  32, 8, 2
+BLOCKCOPY_PP2  32, 12, 2
+BLOCKCOPY_PP2  32, 16, 2
+BLOCKCOPY_PP2  32, 24, 2
+BLOCKCOPY_PP2  32, 32, 2
+BLOCKCOPY_PP2  32, 48, 2
+BLOCKCOPY_PP2  32, 64, 2
+BLOCKCOPY_PP2  48, 64, 4
+BLOCKCOPY_PP2  64, 16, 4
+BLOCKCOPY_PP2  64, 32, 4
+BLOCKCOPY_PP2  64, 48, 4
+BLOCKCOPY_PP2  64, 64, 4
+#endif
+
+//void blockcopy_ss_c(int16_t* a, intptr_t stridea, const int16_t* b, intptr_t strideb)
+.macro BLOCKCOPY_SS bx, by, lmul
+function PFX(blockcopy_ss_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+loop_blockcopy_ss_\bx\()x\by:
+    vle16.v         v0, (a2)
+    vse16.v         v0, (a0)
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_blockcopy_ss_\bx\()x\by
+    ret
+endfunc
+.endm
+
+.macro BLOCKCOPY_SS2 bx, by, lmul
+function PFX(blockcopy_ss_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    slli            a4, a1, 1
+    slli            a6, a3, 1
+    add             a5, a4, a1
+    add             a7, a6, a3
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+loop_blockcopy_ss_\bx\()x\by:
+    add             t3, a2, a3
+    add             t4, a2, a6
+    add             t5, a2, a7
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+    vle16.v         v8, (t4)
+    vle16.v         v12, (t5)
+    add             a2, t5, a3
+    add             t3, a0, a1
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vse16.v         v0, (a0)
+    vse16.v         v4, (t3)
+    vse16.v         v8, (t4)
+    vse16.v         v12, (t5)
+    add             a0, t5, a1
+    addi            t1, t1, -4
+    bgtz            t1, loop_blockcopy_ss_\bx\()x\by
+    ret
+endfunc
+.endm
+
+BLOCKCOPY_SS2  4, 4, 1
+BLOCKCOPY_SS2  4, 8, 1
+BLOCKCOPY_SS2  8, 8, 1
+BLOCKCOPY_SS2  8, 16, 1
+BLOCKCOPY_SS2  16, 16, 2
+BLOCKCOPY_SS2  16, 24, 2
+BLOCKCOPY_SS2  16, 32, 2
+BLOCKCOPY_SS2  32, 32, 4
+BLOCKCOPY_SS2  32, 64, 4
+BLOCKCOPY_SS   64, 64, 8
+
+//void blockcopy_sp_c(pixel* a, intptr_t stridea, const int16_t* b, intptr_t strideb)
+.macro BLOCKCOPY_SP bx, by, lmul, lmul2
+function PFX(blockcopy_sp_\bx\()x\by\()_v)
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+#endif
+    slli            a3, a3, 1
+    li              t1, \by
+    li              t2, \bx
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+loop_blockcopy_sp_\bx\()x\by:
+    vle16.v         v0, (a2)
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (a0)
+#else
+    vsetvli         zero, t2, e8, m\lmul2, ta, ma
+    vnsrl.wi        v8, v0, 0
+    vse8.v          v8, (a0)
+#endif
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_blockcopy_sp_\bx\()x\by
+    ret
+endfunc
+.endm
+
+.macro BLOCKCOPY_SP2 bx, by, lmul, lmul2
+function PFX(blockcopy_sp_\bx\()x\by\()_v)
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+#endif
+    slli            a3, a3, 1
+    slli            a4, a1, 1
+    slli            a6, a3, 1
+    add             a5, a4, a1
+    add             a7, a6, a3
+    li              t1, \by
+    li              t2, \bx
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+loop_blockcopy_sp_\bx\()x\by:
+    add             t3, a2, a3
+    add             t4, a2, a6
+    add             t5, a2, a7
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+    vle16.v         v8, (t4)
+    vle16.v         v12, (t5)
+    add             a2, t5, a3
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (a0)
+#else
+    vsetvli         zero, t2, e8, m\lmul2, ta, ma
+    add             t3, a0, a1
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vnsrl.wi        v16, v0, 0
+    vnsrl.wi        v20, v4, 0
+    vnsrl.wi        v24, v8, 0
+    vnsrl.wi        v28, v12, 0
+    vse8.v          v16, (a0)
+    vse8.v          v20, (t3)
+    vse8.v          v24, (t4)
+    vse8.v          v28, (t5)
+#endif
+    add             a0, t5, a1
+    addi            t1, t1, -4
+    bgtz            t1, loop_blockcopy_sp_\bx\()x\by
+    ret
+endfunc
+.endm
+
+BLOCKCOPY_SP2  4, 4, 1, 1
+BLOCKCOPY_SP2  4, 8, 1, 1
+BLOCKCOPY_SP2  8, 8, 1, 1
+BLOCKCOPY_SP2  8, 16, 1, 1
+BLOCKCOPY_SP2  16, 16, 2, 1
+BLOCKCOPY_SP2  16, 24, 2, 1
+BLOCKCOPY_SP2  16, 32, 2, 1
+BLOCKCOPY_SP2  32, 32, 4, 2
+BLOCKCOPY_SP2  32, 64, 4, 2
+BLOCKCOPY_SP   64, 64, 8, 4
+
+//void blockcopy_ps_c(int16_t* a, intptr_t stridea, const pixel* b, intptr_t strideb)
+.macro BLOCKCOPY_PS bx, by, lmul, lmul2
+function PFX(blockcopy_ps_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+#else
+    vsetvli         zero, t2, e8, m\lmul2, ta, ma
+#endif
+loop_blockcopy_ps_\bx\()x\by:
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vse16.v         v0, (a0)
+#else
+    vsetvli         zero, t2, e8, m\lmul2, ta, ma
+    vle8.v          v0, (a2)
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+    vzext.vf2       v8, v0
+    vse16.v         v8, (a0)
+#endif
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            t1, t1, -1
+    bgtz            t1, loop_blockcopy_ps_\bx\()x\by
+    ret
+endfunc
+.endm
+
+.macro BLOCKCOPY_PS2 bx, by, lmul, lmul2
+function PFX(blockcopy_ps_\bx\()x\by\()_v)
+    li              t1, \by
+    li              t2, \bx
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+#endif
+    slli            a4, a1, 1
+    slli            a6, a3, 1
+    add             a5, a4, a1
+    add             a7, a6, a3
+loop_blockcopy_ps_\bx\()x\by:
+    add             t3, a2, a3
+    add             t4, a2, a6
+    add             t5, a2, a7
+#if HIGH_BIT_DEPTH
+    vle16.v         v0, (a2)
+    vle16.v         v4, (t3)
+    vle16.v         v8, (t4)
+    vle16.v         v12, (t5)
+#else
+    vsetvli         zero, t2, e8, m\lmul2, ta, ma
+    vle8.v          v16, (a2)
+    vle8.v          v20, (t3)
+    vle8.v          v24, (t4)
+    vle8.v          v28, (t5)
+    vsetvli         zero, t2, e16, m\lmul, ta, ma
+    vzext.vf2       v0, v16
+    vzext.vf2       v4, v20
+    vzext.vf2       v8, v24
+    vzext.vf2       v12, v28
+#endif
+    add             a2, t5, a3
+    add             t3, a0, a1
+    add             t4, a0, a4
+    add             t5, a0, a5
+    vse16.v         v0, (a0)
+    vse16.v         v4, (t3)
+    vse16.v         v8, (t4)
+    vse16.v         v12, (t5)
+    add             a0, t5, a1
+    addi            t1, t1, -4
+    bgtz            t1, loop_blockcopy_ps_\bx\()x\by
+    ret
+endfunc
+.endm
+
+BLOCKCOPY_PS2  4, 4, 1, 1
+BLOCKCOPY_PS2  4, 8, 1, 1
+BLOCKCOPY_PS2  8, 8, 1, 1
+BLOCKCOPY_PS2  8, 16, 1, 1
+BLOCKCOPY_PS2  16, 16, 2, 1
+BLOCKCOPY_PS2  16, 24, 2, 1
+BLOCKCOPY_PS2  16, 32, 2, 1
+BLOCKCOPY_PS2  32, 32, 4, 2
+BLOCKCOPY_PS2  32, 64, 4, 2
+BLOCKCOPY_PS   64, 64, 8, 4
+
+//void planecopy_cp_c(const uint8_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift)
+function PFX(planecopy_cp_v)
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+    vsetvli         zero, a4, e16, m2, ta, ma
+#else
+    vsetvli         zero, a4, e8, m1, ta, ma
+#endif
+    vmv.v.x         v30, a6
+
+loop_planecopy_cp_y:
+    mv              t0, a0
+    mv              t1, a2
+    mv              t2, a4
+loop_planecopy_cp_x:
+#if HIGH_BIT_DEPTH
+    vsetvli         t3, t2, e8, m1, ta, ma
+    slli            t4, t3, 1
+    vle8.v          v0, (t0)
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vzext.vf2       v2, v0
+    vsll.vv         v2, v2, v30
+    vse16.v         v2, (t1)
+#else
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v0, (t0)
+    vsll.vv         v2, v0, v30
+    vse8.v          v2, (t1)
+#endif
+    sub             t2, t2, t3
+    add             t0, t0, t3
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t4
+#else
+    add             t1, t1, t3
+#endif
+    bgtz            t2, loop_planecopy_cp_x
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            a5, a5, -1
+    bgtz            a5, loop_planecopy_cp_y
+    ret
+endfunc
+
+//void planecopy_sp_c(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask)
+function PFX(planecopy_sp_v)
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+#endif
+    vsetvli         zero, a4, e16, m1, ta, ma
+    vmv.v.x         v30, a6
+    vmv.v.x         v28, a7
+
+loop_planecopy_sp_y:
+    mv              t0, a0
+    mv              t1, a2
+    mv              t2, a4
+loop_planecopy_sp_x:
+    vsetvli         t3, t2, e16, m1, ta, ma
+    slli            t4, t3, 1
+    vle16.v         v0, (t0)
+    vsrl.vv         v0, v0, v30
+    vand.vv         v0, v0, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (t1)
+#else
+    vsetvli         zero, t3, e8, m1, ta, ma
+    vnsrl.wi        v2, v0, 0
+    vse8.v          v2, (t1)
+#endif
+    sub             t2, t2, t3
+    add             t0, t0, t4
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t4
+#else
+    add             t1, t1, t3
+#endif
+    bgtz            t2, loop_planecopy_sp_x
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            a5, a5, -1
+    bgtz            a5, loop_planecopy_sp_y
+    ret
+endfunc
+
+//void planecopy_pp_shr_c(const pixel* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift)
+function PFX(planecopy_pp_shr_v)
+#if HIGH_BIT_DEPTH
+    slli            a1, a1, 1
+    slli            a3, a3, 1
+    vsetvli         zero, a4, e16, m1, ta, ma
+#else
+    vsetvli         zero, a4, e8, m1, ta, ma
+#endif
+    vmv.v.x         v30, a6
+
+loop_planecopy_pp_shr_y:
+    mv              t0, a0
+    mv              t1, a2
+    mv              t2, a4
+loop_planecopy_pp_shr_x:
+#if HIGH_BIT_DEPTH
+    vsetvli         t3, t2, e16, m1, ta, ma
+    slli            t4, t3, 1
+    vle16.v         v0, (t0)
+    vsrl.vv         v0, v0, v30
+    vse16.v         v0, (t1)
+#else
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v0, (t0)
+    vsrl.vv         v0, v0, v30
+    vse8.v          v0, (t1)
+#endif
+    sub             t2, t2, t3
+#if HIGH_BIT_DEPTH
+    add             t0, t0, t4
+    add             t1, t1, t4
+#else
+    add             t0, t0, t3
+    add             t1, t1, t3
+#endif
+    bgtz            t2, loop_planecopy_pp_shr_x
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            a5, a5, -1
+    bgtz            a5, loop_planecopy_pp_shr_y
+    ret
+endfunc
+
+//void planecopy_sp_shl_c(const uint16_t* src, intptr_t srcStride, pixel* dst, intptr_t dstStride, int width, int height, int shift, uint16_t mask)
+function PFX(planecopy_sp_shl_v)
+    slli            a1, a1, 1
+#if HIGH_BIT_DEPTH
+    slli            a3, a3, 1
+#endif
+    vsetvli         zero, a4, e16, m1, ta, ma
+    vmv.v.x         v30, a6
+    vmv.v.x         v28, a7
+
+loop_planecopy_sp_shl_y:
+    mv              t0, a0
+    mv              t1, a2
+    mv              t2, a4
+loop_planecopy_sp_shl_x:
+    vsetvli         t3, t2, e16, m1, ta, ma
+    slli            t4, t3, 1
+    vle16.v         v0, (t0)
+    vsll.vv         v0, v0, v30
+    vand.vv         v0, v0, v28
+#if HIGH_BIT_DEPTH
+    vse16.v         v0, (t1)
+#else
+    vsetvli         zero, t3, e8, m1, ta, ma
+    vnsrl.wi        v2, v0, 0
+    vse8.v          v2, (t1)
+#endif
+    sub             t2, t2, t3
+    add             t0, t0, t4
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t4
+#else
+    add             t1, t1, t3
+#endif
+    bgtz            t2, loop_planecopy_sp_shl_x
+    add             a0, a0, a1
+    add             a2, a2, a3
+    addi            a5, a5, -1
+    bgtz            a5, loop_planecopy_sp_shl_y
+    ret
+endfunc
+
+// uint32_t costCoeffNxN(
+//    uint16_t *scan,        // a0
+//    coeff_t *coeff,        // a1
+//    intptr_t trSize,       // a2
+//    uint16_t *absCoeff,    // a3
+//    uint8_t *tabSigCtx,    // a4
+//    uint16_t scanFlagMask, // a5
+//    uint8_t *baseCtx,      // a6
+//    int offset,            // a7
+//    int scanPosSigOff,     // sp
+//    int subPosBase)        // sp + 8, or sp + 4 on APPLE
+function PFX(costCoeffNxN_rvv)
+    // load coeff
+    slli            a2, a2, 1
+    vsetivli        zero, 4, e16, m1, ta, ma
+    add             t0, a1, a2
+    add             t1, t0, a2
+    add             t2, t1, a2
+    vle16.v         v16, (a1)
+    vle16.v         v18, (t0)
+    vle16.v         v17, (t1)
+    vle16.v         v19, (t2)
+
+    vsetivli        zero, 16, e16, m1, ta, ma
+    vslideup.vi     v16, v18, 4
+    vslideup.vi     v17, v19, 4
+    vslideup.vi     v16, v17, 8
+
+    // loading tabSigCtx
+    vsetivli        zero, 16, e8, m1, ta, ma
+    vle8.v          v20, (a4)
+
+    vsetivli        zero, 16, e16, m2, ta, ma
+    vzext.vf2       v12, v20
+    VABS            v6, v16, v30
+
+    // load scan table
+    lw              a2, 0(sp)
+    xori            t6, a2, 15
+    slli            t0, t6, 1
+    add             t1, a0, t0
+    vle16.v         v8, (t1)
+    vmv.v.i         v10, 15
+    vxor.vv         v4, v8, v10
+
+    // reorder abs(coeff)
+    vrgatherei16.vv  v2, v6, v4
+
+    // reorder tabSigCtx
+    vrgatherei16.vv  v14, v12, v4
+    vadd.vx          v0, v14, a7
+
+    // register mapping
+    // a0 - sum
+    // a1 - entropyStateBits
+    // {v0, v1} - sigCtx
+    // {v2, v3} - abs(coeff)
+    // a2 - scanPosSigOff
+    // a3 - absCoeff
+    // a4 - numNonZero
+    // a5 - scanFlagMask
+    // a6 - baseCtx
+    // a7 - scanPosSigOff & 15
+    mv              a7, t6
+    li              a0, 0
+    la              a1, PFX(entropyStateBits)
+    li              a4, 0
+    beqz            a2, idx_zero
+
+loop_nonzero:
+    addi            a2, a2, -1
+    slli            t0, a4, 1
+    add             t0, a3, t0
+    vmv.x.s         t1, v2
+    vmv.x.s         t6, v0              // t6 = ctxSig
+    vslidedown.vi   v2, v2, 1
+    vslidedown.vi   v0, v0, 1
+    sh              t1, 0(t0)           // absCoeffnumNonZero = tmpCoeffblkPos
+
+    andi            t5, a5, 1           // t5  sig = scanFlagMask & 1
+    srli            a5, a5, 1           // scanFlagMask >>= 1;
+    add             a4, a4, t5          // numNonZero += sig
+
+    add             t0, a6, t6
+    lbu             t4, 0(t0)           // mstate = baseCtxctxSig load uint8
+    andi            t3, t4, 1           // t3 mps = mstate & 1
+    xor             t4, t4, t5          // t4 = mstate ^ sig
+
+    slli            t0, t4, 2
+    add             t0, a1, t0
+    lw              t1, 0(t0)           // stateBits = PFX(entropyStateBits)mstate ^ sig
+    add             a0, a0, t1          // sum += stateBits
+    srli            t1, t1, 24
+    add             t2, t1, t3          // t2 nextState = (stateBits >> 24) + mps
+
+    li              t0, 1
+    bne             t4, t0, 1f          // if ((mstate ^ sig) == 1)
+    mv              t2, t5              // nextState = sig
+1:
+    add             t0, a6, t6
+    sb              t2, 0(t0)           // baseCtxctxSig = nextState
+    bnez            a2, loop_nonzero
+
+idx_zero:
+    slli            t0, a4, 1
+    add             t0, a3, t0
+    vmv.x.s         t1, v2
+    vmv.x.s         t6, v0
+    sh              t1, 0(t0)           // absCoeffnumNonZero = tmpCoeffblkPos
+
+    lw              t0, 8(sp)
+    li              t1, 0xFFFF
+    and             t0, t0, t1          // subPosBase &= 0xFFFF
+    seqz            a2, t0
+
+    add             a4, a4, a7
+    add             a4, a4, a2
+    beqz            a4, 2f              // subPosBase != 0 && numNonZero == 0,exit
+
+    addi            a2, a2, -1
+    and             t6, t6, a2          // ctxSig &= posZeroMask
+
+    andi            t5, a5, 1           // t5, sig = scanFlagMask & 1
+
+    add             t0, a6, t6
+    lbu             t4, 0(t0)           // mstate = baseCtxctxSig load uint8
+    andi            t3, t4, 1           // t3 mps = mstate & 1
+    xor             t4, t4, t5          // t4 = mstate ^ sig
+
+    slli            t0, t4, 2
+    add             t0, a1, t0
+    lw              t1, 0(t0)           // stateBits = PFX(entropyStateBits)mstate ^ sig
+    add             a0, a0, t1          // sum += stateBits
+    srli            t1, t1, 24
+    add             t2, t1, t3          // t2 nextState = (stateBits >> 24) + mps
+
+    li              t0, 1
+    bne             t4, t0, 3f          // if ((mstate ^ sig) == 1)
+    mv              t2, t5              // nextState = sig
+3:
+    add             t0, a6, t6
+    sb              t2, 0(t0)           // baseCtxctxSig = nextState
+2:
+    li              t0, 0xFFFFFF
+    and             a0, a0, t0          //sum &=0xFFFFFF
+    ret
+endfunc

x265-4.2.tar/source/common/riscv64/riscv64_utils.S Added

@@ -0,0 +1,212 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+.align 4
+
+.text
+
+.macro CALC_STRIDED_ADDRS src, stride, reg1, reg2, reg3, reg4
+    slli                 \reg4, \stride, 1          # reg4 = stride * 2
+    add                  \reg1, \src, \stride       # reg1 = src + stride
+    add                  \reg2, \src, \reg4         # reg2 = src + 2*stride
+    add                  \reg3, \reg2, \stride      # reg3 = src + 3*stride
+.endm
+
+.macro LOAD_4_LINE src, es, sstride, d1, d2, d3, d4, reg1, reg2, reg3, reg4
+    CALC_STRIDED_ADDRS   \src, \sstride, \reg1, \reg2, \reg3, \reg4
+    vl\es\().v           \d1, (\src)
+    vl\es\().v           \d2, (\reg1)
+    vl\es\().v           \d3, (\reg2)
+    vl\es\().v           \d4, (\reg3)
+.endm
+
+.macro STORE_4_LINE dst, es, dstride, d1, d2, d3, d4, reg1, reg2, reg3, reg4
+    CALC_STRIDED_ADDRS   \dst, \dstride, \reg1, \reg2, \reg3, \reg4
+    vs\es\().v           \d1, (\dst)
+    vs\es\().v           \d2, (\reg1)
+    vs\es\().v           \d3, (\reg2)
+    vs\es\().v           \d4, (\reg3)
+.endm
+
+.macro TRANSPOSE_HELPER a, b, c, d, e, f, g, h, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7, tmp8
+    vslideup.vi         \tmp1, \b, 1
+    vslidedown.vi       \tmp2, \a, 1
+    vslideup.vi         \tmp3, \d, 1
+    vslidedown.vi       \tmp4, \c, 1
+    vslideup.vi         \tmp5, \f, 1
+    vslidedown.vi       \tmp6, \e, 1
+    vslideup.vi         \tmp7, \h, 1
+    vslidedown.vi       \tmp8, \g, 1
+
+    vmerge.vvm          \a, \tmp1, \a, v0
+    vmerge.vvm          \b, \b, \tmp2, v0
+
+    vmerge.vvm          \c, \tmp3, \c, v0
+    vmerge.vvm          \d, \d, \tmp4, v0
+
+    vmerge.vvm          \e, \tmp5, \e, v0
+    vmerge.vvm          \f, \f, \tmp6, v0
+
+    vmerge.vvm          \g, \tmp7, \g, v0
+    vmerge.vvm          \h, \h, \tmp8, v0
+.endm
+
+#if !HIGH_BIT_DEPTH
+# void transpose4x4_rvv(uint8_t *dst, const uint8_t *src, intptr_t dstride, intptr_t sstride)
+function transpose4x4_rvv_u8
+    li                    t4, 0x5
+    vsetivli              zero, 4, e8, mf2, ta, ma
+    vmv.v.x               v0, t4
+
+    LOAD_4_LINE           a1, e8, a3, v4, v5, v6, v7, t0, t1, t2, t3
+    TRANSPOSE_4x4         8, v16, v17, v18, v19, v4, v5, v6, v7
+
+    vsetivli              zero, 4, e8, mf2, ta, ma
+    STORE_4_LINE          a0, e8, a2, v16, v17, v18, v19, t0, t1, t2, t3
+
+    ret
+endfunc
+
+function transpose8x8_rvv_u8
+    li                    t4, 0x55
+    vsetivli              zero, 8, e8, m1, ta, ma
+    vmv.v.x               v0, t4
+
+    LOAD_4_LINE           a1, e8, a3, v4, v5, v6, v7, t0, t1, t2, t3
+    add                   t4, t2, a3
+    LOAD_4_LINE           t4, e8, a3, v8, v9, v10 v11, t0, t1, t2, t3
+
+    TRANSPOSE_8x8         8, v16, v17, v18, v19, v20, v21, v22, v23, v4, v5, v6, v7, v8, v9, v10, v11
+
+    vsetivli              zero, 8, e8, m1, ta, ma
+    STORE_4_LINE          a0, e8, a2, v16, v17, v18, v19, t0, t1, t2, t3
+    add                   t4, t2, a2
+    STORE_4_LINE          t4, e8, a2, v20, v21, v22, v23, t0, t1, t2, t3
+
+    ret
+endfunc
+
+function transpose16x16_rvv_u8
+    # v0 must be 0x5555 as mask
+    li                   t4, 0x5555
+    vsetivli             zero, 16, e8, m1, ta, ma
+    vmv.v.x              v0, t4
+
+    # load: v1-v16
+    LOAD_4_LINE           a1, e8, a3, v1, v2, v3, v4, t0, t1, t2, t3
+    add                   t4, t2, a3
+    LOAD_4_LINE           t4, e8, a3, v5, v6, v7, v8, t0, t1, t2, t3
+    add                   t4, t2, a3
+    LOAD_4_LINE           t4, e8, a3, v9, v10, v11, v12, t0, t1, t2, t3
+    add                   t4, t2, a3
+    LOAD_4_LINE           t4, e8, a3, v13, v14, v15, v16, t0, t1, t2, t3
+
+    # ------------------------------------------------------------
+    # Stage 1: e8, VL=16
+    # ------------------------------------------------------------
+    TRANSPOSE_HELPER      v1, v2, v3, v4, v5, v6, v7, v8, v17, v18, v19, v20, v21, v22, v23, v24
+    TRANSPOSE_HELPER      v9, v10, v11, v12, v13, v14, v15, v16, v17, v18, v19, v20, v21, v22, v23, v24
+
+    # ------------------------------------------------------------
+    # Stage 2:  e16, VL=8
+    # ------------------------------------------------------------
+    vsetivli             zero, 8, e16, m1, ta, ma
+
+    TRANSPOSE_HELPER     v1, v3, v2, v4, v5, v7, v6, v8, v17, v18, v19, v20, v21, v22, v23, v24
+    TRANSPOSE_HELPER     v9, v11, v10, v12, v13, v15, v14, v16, v17, v18, v19, v20, v21, v22, v23, v24
+
+
+    # ------------------------------------------------------------
+    # Stage 3: e32, VL=4
+    # ------------------------------------------------------------
+    vsetivli             zero, 4, e32, m1, ta, ma
+
+    TRANSPOSE_HELPER     v1, v5, v2, v6, v3, v7, v4, v8, v17, v18, v19, v20, v21, v22, v23, v24
+    TRANSPOSE_HELPER     v9, v13, v10, v14, v11, v15, v12, v16, v17, v18, v19, v20, v21, v22, v23, v24
+
+
+    # ------------------------------------------------------------
+    # Stage 4: e64, VL=2
+    # ------------------------------------------------------------
+    vsetivli             zero, 2, e64, m1, ta, ma
+
+    TRANSPOSE_HELPER     v1, v9, v2, v10, v3, v11, v4, v12, v17, v18, v19, v20, v21, v22, v23, v24
+    TRANSPOSE_HELPER     v5, v13, v6, v14, v7, v15, v8, v16, v17, v18, v19, v20, v21, v22, v23, v24
+
+
+    # store: v1-v16
+    vsetivli              zero, 16, e8, m1, ta, ma
+    STORE_4_LINE          a0, e8, a2, v1, v2, v3, v4, t0, t1, t2, t3
+    add                   t4, t2, a2
+    STORE_4_LINE          t4, e8, a2, v5, v6, v7, v8, t0, t1, t2, t3
+    add                   t4, t2, a2
+    STORE_4_LINE          t4, e8, a2, v9, v10, v11, v12, t0, t1, t2, t3
+    add                   t4, t2, a2
+    STORE_4_LINE          t4, e8, a2, v13, v14, v15, v16, t0, t1, t2, t3
+
+    ret
+endfunc
+
+#else
+# void transpose4x4_rvv(uint16_t *dst, const uint16_t *src, intptr_t dstride, intptr_t sstride)
+function transpose4x4_rvv_u16
+    li                     t4, 0x5
+    vsetivli               zero, 4, e16, mf2, ta, ma
+    vmv.v.x                v0, t4
+
+    slli                   t5, a3, 1
+    slli                   t6, a2, 1
+
+    LOAD_4_LINE            a1, e16, t5, v4, v5, v6, v7, t0, t1, t2, t3
+    TRANSPOSE_4x4          16, v16, v17, v18, v19, v4, v5, v6, v7
+
+    vsetivli               zero, 4, e16, mf2, ta, ma
+    STORE_4_LINE           a0, e16, t6, v16, v17, v18, v19, t0, t1, t2, t3
+
+    ret
+endfunc
+
+function transpose8x8_rvv_u16
+    li                    t4, 0x55
+    vsetivli              zero, 8, e16, m1, ta, ma
+    vmv.v.x               v0, t4
+
+    slli                  t5, a3, 1
+    slli                  t6, a2, 1
+
+    LOAD_4_LINE           a1, e16, t5, v4, v5, v6, v7, t0, t1, t2, t3
+    add                   t4, t2, t5
+    LOAD_4_LINE           t4, e16, t5, v8, v9, v10 v11, t0, t1, t2, t3
+    TRANSPOSE_8x8         16, v16, v17, v18, v19, v20, v21, v22, v23, v4, v5, v6, v7, v8, v9, v10, v11
+
+    vsetivli              zero, 8, e16, m1, ta, ma
+    STORE_4_LINE          a0, e16, t6, v16, v17, v18, v19, t0, t1, t2, t3
+    add                   t4, t2, t6
+    STORE_4_LINE          t4, e16, t6, v20, v21, v22, v23, t0, t1, t2, t3
+
+    ret
+endfunc
+
+#endif
\ No newline at end of file

x265-4.2.tar/source/common/riscv64/riscv64_utils.cpp Added

@@ -0,0 +1,111 @@
+
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "common.h"
+#include "x265.h"
+#include "riscv64_utils.h"
+#include <riscv_vector.h>
+
+namespace X265_NS
+{
+
+#if !HIGH_BIT_DEPTH
+void transpose32x32_rvv(uint8_t *dst, const uint8_t *src, intptr_t dstride, intptr_t sstride)
+{
+    transpose16x16_rvv(dst, src, dstride, sstride);
+    transpose16x16_rvv(dst + 16 * dstride + 16, src + 16 * sstride + 16, dstride, sstride);
+    if (dst == src)
+    {
+        size_t vl = 16;
+        uint8_t tmp16 * 16 __attribute__((aligned(64)));
+        transpose16x16_rvv(tmp, src + 16, 16, sstride);
+        transpose16x16_rvv(dst + 16, src + 16 * sstride, dstride, sstride);
+        for (int i = 0; i < 16; i++)
+        {
+            __riscv_vse8_v_u8m1(dst + (16 + i) * dstride, __riscv_vle8_v_u8m1(tmp + 16 * i, vl), vl);
+        }
+    }
+    else
+    {
+        transpose16x16_rvv(dst + 16 * dstride, src + 16, dstride, sstride);
+        transpose16x16_rvv(dst + 16, src + 16 * sstride, dstride, sstride);
+    }
+
+}
+
+#else
+void transpose16x16_rvv(uint16_t *dst, const uint16_t *src, intptr_t dstride, intptr_t sstride)
+{
+    transpose8x8_rvv(dst, src, dstride, sstride);
+    transpose8x8_rvv(dst + 8 * dstride + 8, src + 8 * sstride + 8, dstride, sstride);
+
+    if (dst == src)
+    {
+        uint16_t tmp8 * 8;
+        size_t vl = 8;
+        transpose8x8_rvv(tmp, src + 8, 8, sstride);
+        transpose8x8_rvv(dst + 8, src + 8 * sstride, dstride, sstride);
+        for (int i = 0; i < 8; i++)
+        {
+            __riscv_vse16_v_u16m1(dst + (8 + i) * dstride, __riscv_vle16_v_u16m1(tmp + 8 * i, vl), vl);
+        }
+    }
+    else
+    {
+        transpose8x8_rvv(dst + 8 * dstride, src + 8, dstride, sstride);
+        transpose8x8_rvv(dst + 8, src + 8 * sstride, dstride, sstride);
+    }
+
+}
+
+void transpose32x32_rvv(uint16_t *dst, const uint16_t *src, intptr_t dstride, intptr_t sstride)
+{
+    //assumption: there is no partial overlap
+    for (int i = 0; i < 4; i++)
+    {
+        transpose8x8_rvv(dst + i * 8 * (1 + dstride), src + i * 8 * (1 + sstride), dstride, sstride);
+        for (int j = i + 1; j < 4; j++)
+        {
+            if (dst == src)
+            {
+                uint16_t tmp8 * 8 __attribute__((aligned(64)));
+                size_t vl = 8;
+                transpose8x8_rvv(tmp, src + 8 * i + 8 * j * sstride, 8, sstride);
+                transpose8x8_rvv(dst + 8 * i + 8 * j * dstride, src + 8 * j + 8 * i * sstride, dstride, sstride);
+                for (int k = 0; k < 8; k++)
+                {
+                    __riscv_vse16_v_u16m1(dst + 8 * j + (8 * i + k) * dstride, __riscv_vle16_v_u16m1(tmp + 8 * k, vl), vl);
+                }
+            }
+            else
+            {
+                transpose8x8_rvv(dst + 8 * (j + i * dstride), src + 8 * (i + j * sstride), dstride, sstride);
+                transpose8x8_rvv(dst + 8 * (i + j * dstride), src + 8 * (j + i * sstride), dstride, sstride);
+            }
+
+        }
+    }
+}
+#endif // !HIGH_BIT_DEPTH
+}
\ No newline at end of file

x265-4.2.tar/source/common/riscv64/riscv64_utils.h Added

@@ -0,0 +1,69 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef __RISCV64_UTILS_H__
+#define __RISCV64_UTILS_H__
+
+#include <stdint.h>
+
+extern "C" {
+    void transpose4x4_rvv_u8(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride);
+    void transpose8x8_rvv_u8(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride);
+
+    void transpose4x4_rvv_u16(uint16_t* dst, const uint16_t* src, intptr_t dstride, intptr_t sstride);
+    void transpose8x8_rvv_u16(uint16_t* dst, const uint16_t* src, intptr_t dstride, intptr_t sstride);
+
+    void transpose16x16_rvv_u8(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride);
+}
+
+namespace X265_NS
+{
+
+// overload
+inline void transpose4x4_rvv(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride) {
+    transpose4x4_rvv_u8(dst, src, dstride, sstride);
+}
+
+inline void transpose4x4_rvv(uint16_t* dst, const uint16_t* src, intptr_t dstride, intptr_t sstride) {
+    transpose4x4_rvv_u16(dst, src, dstride, sstride);
+}
+
+inline void transpose8x8_rvv(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride) {
+    transpose8x8_rvv_u8(dst, src, dstride, sstride);
+}
+
+inline void transpose8x8_rvv(uint16_t* dst, const uint16_t* src, intptr_t dstride, intptr_t sstride) {
+    transpose8x8_rvv_u16(dst, src, dstride, sstride);
+}
+
+inline void transpose16x16_rvv(uint8_t* dst, const uint8_t* src, intptr_t dstride, intptr_t sstride) {
+    transpose16x16_rvv_u8(dst, src, dstride, sstride);
+}
+
+void transpose32x32_rvv(uint8_t *dst, const uint8_t *src, intptr_t dstride, intptr_t sstride);
+
+void transpose16x16_rvv(uint16_t *dst, const uint16_t *src, intptr_t dstride, intptr_t sstride);
+void transpose32x32_rvv(uint16_t *dst, const uint16_t *src, intptr_t dstride, intptr_t sstride);
+}
+
+#endif
\ No newline at end of file

x265-4.2.tar/source/common/riscv64/sad-a.S Added

@@ -0,0 +1,940 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Jia Yuan <yuan.jia@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+.align 4
+
+.text
+
+.macro push
+    addi          sp, sp, -48
+    sd            ra, 16(sp)    // save return address
+    sd            s0, 32(sp)    // save the frame pointer
+    addi          s0, sp, 48    // set the new frame pointer
+
+    sd            x9, 0(sp)
+.endm
+
+.macro  pop
+    ld           x9, 0(sp)
+
+    // restore the frame pointer and return address
+    ld           s0, 32(sp)
+    ld           ra, 16(sp)
+    addi         sp, sp, 48
+.endm
+
+#if !HIGH_BIT_DEPTH
+.macro SAD_FUNC_SMALL w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li            t0, \h
+    li            t1, \w
+    // prevent overflow when there is a bigger \h
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v28, 0
+
+1:
+    addi          t0, t0, -4
+    vsetvli       zero, t1, e8, m\()\lmul1, ta, ma
+
+    vle8.v        v0, (a0)
+    vle8.v        v1, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v8, v0, v1, v24
+
+    vle8.v        v2, (a0)
+    vle8.v        v3, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v9, v2, v3, v25
+
+    vle8.v        v4, (a0)
+    vle8.v        v5, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v10, v4, v5, v26
+
+    vle8.v        v6, (a0)
+    vle8.v        v7, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v11, v6, v7, v27
+
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+
+    vzext.vf4     v0, v8
+    vzext.vf4     v4, v9
+    vadd.vv       v28, v28, v0
+    vadd.vv       v28, v28, v4
+
+    vzext.vf4     v12, v10
+    vzext.vf4     v16, v11
+    vadd.vv       v28, v28, v12
+    vadd.vv       v28, v28, v16
+
+    bnez          t0, 1b
+
+    vmv.v.i       v20, 0
+    vredsum.vs    v20, v28, v20
+    vmv.x.s       a0, v20
+    ret
+endfunc
+.endm
+
+SAD_FUNC_SMALL 4, 4, 1, 1
+SAD_FUNC_SMALL 4, 8, 1, 1
+SAD_FUNC_SMALL 4, 16, 1, 1
+
+SAD_FUNC_SMALL 8, 4, 1, 2
+SAD_FUNC_SMALL 8, 8, 1, 2
+SAD_FUNC_SMALL 8, 16, 1, 2
+SAD_FUNC_SMALL 8, 32, 1, 2
+
+SAD_FUNC_SMALL 12, 16, 1, 4
+
+SAD_FUNC_SMALL 16, 4, 1, 4
+SAD_FUNC_SMALL 16, 8, 1, 4
+SAD_FUNC_SMALL 16, 12, 1, 4
+SAD_FUNC_SMALL 16, 16, 1, 4
+SAD_FUNC_SMALL 16, 32, 1, 4
+SAD_FUNC_SMALL 16, 64, 1, 4
+
+.macro SAD_FUNC_MEDIUM w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li           t0, \h
+    li           t1, \w
+
+    vsetvli      zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i      v24, 0
+
+1:
+    addi          t0, t0, -2
+    vsetvli       zero, t1, e8, m\()\lmul1, ta, ma
+
+    vle8.v        v0, (a0)
+    vle8.v        v2, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v8, v0, v2, v12
+
+    vle8.v        v4, (a0)
+    vle8.v        v6, (a2)
+    add           a0, a0, a1
+    add           a2, a2, a3
+    VABDU         v10, v4, v6, v14
+
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+
+    vzext.vf4     v0, v8
+    vadd.vv       v24, v24, v0
+
+    vzext.vf4     v16, v10
+    vadd.vv       v24, v24, v16
+
+    bnez          t0, 1b
+
+    vmv.v.i       v0, 0
+    vredsum.vs    v0, v24, v0
+    vmv.x.s       a0, v0
+    ret
+endfunc
+.endm
+
+SAD_FUNC_MEDIUM  24, 32, 2, 8
+
+SAD_FUNC_MEDIUM  32, 8, 2, 8
+SAD_FUNC_MEDIUM  32, 16, 2, 8
+SAD_FUNC_MEDIUM  32, 24, 2, 8
+SAD_FUNC_MEDIUM  32, 32, 2, 8
+SAD_FUNC_MEDIUM  32, 64, 2, 8
+
+.macro SAD_FUNC_COMMON w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li             t0, \h
+    li             t1, \w
+
+    vsetvli        zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v24, 0
+
+1:  // row loop
+    mv             t3, a0
+    mv             t4, a2
+    addi           t0, t0, -1
+
+2:  // inline segment loop
+    vsetvli         t5, t1, e8, m\()\lmul1, ta, ma
+
+    vle8.v          v0, (t3)
+    vle8.v          v4, (t4)
+    add             t3, t3, t5
+    add             t4, t4, t5
+    VABDU           v8, v0, v4, v12
+
+    vsetvli         zero, t1, e32,m\()\lmul2, tu, ma
+    vzext.vf4       v16, v8
+    vadd.vv         v24, v24, v16
+
+    sub             t1, t1, t5
+    bgtz            t1, 2b
+
+    // update row pointer
+    add             a0, a0, a1
+    add             a2, a2, a3
+
+    li              t1, \w      // reset the inline element counter
+    bnez            t0, 1b
+
+    li              t1, \w
+    vsetvli         zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i         v0, 0
+    vredsum.vs      v0, v24, v0
+    vmv.x.s         a0, v0
+    ret
+endfunc
+.endm
+SAD_FUNC_COMMON  48, 64, 1, 4
+
+.macro SAD_FUNC_LARGE w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li             t0, \h
+    li             t1, \w
+    li             t2, 0
+
+1:
+    addi           t0, t0, -1
+    vsetvli        zero, t1, e8, m\()\lmul1, ta, ma
+
+    vle8.v         v0, (a0)
+    vle8.v         v4, (a2)
+    add            a0, a0, a1
+    add            a2, a2, a3
+    UWABDU         v8, v0, v4, v12, v28
+
+    vsetvli        zero, t1, e16, m\()\lmul2, ta, ma
+    vmv.v.i        v24, 0
+    vredsum.vs     v24, v8, v24
+    vmv.x.s        t3, v24
+    add            t2, t2, t3
+
+    bgtz           t0, 1b
+    mv             a0, t2
+    ret
+endfunc
+.endm
+
+SAD_FUNC_LARGE  64, 16, 4, 8
+SAD_FUNC_LARGE  64, 32, 4, 8
+SAD_FUNC_LARGE  64, 48, 4, 8
+SAD_FUNC_LARGE  64, 64, 4, 8
+
+//============= SAD_X3 and SAD_X4 code start========================
+// static void x264_pixel_sad_x3_##size(pixel *fenc, pixel *pix0, pixel *pix1, pixel *pix2, intptr_t i_stride, int scores3)
+// static void x264_pixel_sad_x4_##size(pixel *fenc, pixel *pix0, pixel *pix1,pixel *pix2, pixel *pix3, intptr_t i_stride, int scores4)
+.macro SAD_X_FUNC_SMALL x, w, h, lmul1, lmul2
+function PFX(sad_x\x\()_\w\()x\h\()_rvv)
+// Make function arguments for x == 3 look like x == 4.
+.if \x == 3
+    mv            t0, a4
+    mv            t1, a5
+    mv            a5, t0                // a5 = i_stride
+    mv            a6, t1                // a6 = scores
+.endif
+    li            t2, FENC_STRIDE
+    li            t3, \w
+    li            t4, \h
+
+    vsetvli       zero, t3, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v16, 0
+    vmv.v.i       v20, 0
+    vmv.v.i       v24, 0
+.if \x == 4
+    vmv.v.i       v28, 0
+.endif
+
+1:
+    addi           t4, t4, -1
+    vsetvli        zero, t3, e8, m\()\lmul1, ta, ma
+
+    vle8.v         v0, (a0)
+    vle8.v         v1, (a1)
+    add            a0, a0, t2
+    add            a1, a1, a5
+    VABDU          v5, v0, v1, v9
+
+    vle8.v         v2, (a2)
+    vle8.v         v3, (a3)
+    add            a2, a2, a5
+    add            a3, a3, a5
+    VABDU          v6, v0, v2, v10
+    VABDU          v7, v0, v3, v11
+.if \x == 4
+    vle8.v         v4, (a4)
+    add            a4, a4, a5
+    VABDU          v8, v0, v4, v12
+.endif
+
+    vsetvli        zero, t3, e32, m\()\lmul2, ta, ma
+
+    vzext.vf4      v12, v5
+    vadd.vv        v16, v16, v12
+
+    vzext.vf4      v0, v6
+    vadd.vv        v20, v20, v0
+
+    vzext.vf4      v4, v7
+    vadd.vv        v24, v24, v4
+.if \x == 4
+    vzext.vf4      v12, v8
+    vadd.vv        v28, v28, v12
+.endif
+
+    bnez           t4, 1b
+
+    // reduce and store the result
+    vsetvli        zero, t3, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v0, 0
+    vredsum.vs     v0, v16, v0
+    vmv.x.s        t0, v0
+    sw             t0, 0(a6)      // scores0
+
+    vmv.v.i        v4, 0
+    vredsum.vs     v4, v20, v4
+    vmv.x.s        t1, v4
+    sw             t1, 4(a6)      // scores1
+
+    vmv.v.i        v8, 0
+    vredsum.vs     v8, v24, v8
+    vmv.x.s        t5, v8
+    sw             t5, 8(a6)      // scores2
+
+.if \x == 4
+    vmv.v.i       v12, 0
+    vredsum.vs    v12, v28, v12
+    vmv.x.s       t6, v12
+
+    sw            t6, 12(a6)      // scores3
+.endif
+    ret
+endfunc
+.endm
+
+SAD_X_FUNC_SMALL 3, 4, 4, 1, 1
+SAD_X_FUNC_SMALL 3, 4, 8, 1, 1
+SAD_X_FUNC_SMALL 3, 4, 16, 1, 1
+
+SAD_X_FUNC_SMALL 3, 8, 4, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 8, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 16, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 32, 1, 2
+
+SAD_X_FUNC_SMALL 4, 4, 4, 1, 1
+SAD_X_FUNC_SMALL 4, 4, 8, 1, 1
+SAD_X_FUNC_SMALL 4, 4, 16, 1, 1
+SAD_X_FUNC_SMALL 4, 8, 4, 1, 2
+
+SAD_X_FUNC_SMALL 4, 8, 8, 1, 2
+SAD_X_FUNC_SMALL 4, 8, 16, 1, 2
+SAD_X_FUNC_SMALL 4, 8, 32, 1, 2
+
+SAD_X_FUNC_SMALL 3, 12, 16, 1, 4
+
+SAD_X_FUNC_SMALL 3, 16, 4, 1, 4
+SAD_X_FUNC_SMALL 3, 16, 8, 1, 4
+SAD_X_FUNC_SMALL 3, 16, 12, 1, 4
+SAD_X_FUNC_SMALL 3, 16, 16, 1, 4
+SAD_X_FUNC_SMALL 3, 16, 32, 1, 4
+SAD_X_FUNC_SMALL 3, 16, 64, 1, 4
+
+SAD_X_FUNC_SMALL 4, 12, 16, 1, 4
+
+SAD_X_FUNC_SMALL 4, 16, 4, 1, 4
+SAD_X_FUNC_SMALL 4, 16, 8, 1, 4
+SAD_X_FUNC_SMALL 4, 16, 12, 1, 4
+SAD_X_FUNC_SMALL 4, 16, 16, 1, 4
+SAD_X_FUNC_SMALL 4, 16, 32, 1, 4
+SAD_X_FUNC_SMALL 4, 16, 64, 1, 4
+
+.macro SAD_X_FUNC_COMMON, x, w, h, lmul1, lmul2
+function PFX(sad_x\x\()_\w\()x\h\()_rvv)
+// make function arguments for x == 3 look like x == 4.
+.if \x == 3
+    mv             t0, a4
+    mv             t1, a5
+    mv             a5, t0
+    mv             a6, t1
+.endif
+
+    li             t5, \w
+    li             t6, \h
+
+    vsetvli       zero, t5, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v16, 0
+    vmv.v.i       v20, 0
+    vmv.v.i       v24, 0
+.if \x == 4
+    vmv.v.i       v28, 0
+.endif
+
+1:  // row loop
+    mv             t0, a0   // current fenc
+    mv             t1, a1   // current pix0
+    mv             t2, a2   // current pix1
+    mv             t3, a3   // current pix2
+.if \x == 4
+    mv             t4, a4   // current pix3
+.endif
+    addi           t6, t6, -1
+
+2:  // inline segment loop
+    vsetvli        a7, t5, e8, m\()\lmul1, ta, ma
+
+    vle8.v         v0, (t0)
+    vle8.v         v1, (t1)
+    add            t0, t0, a7
+    add            t1, t1, a7
+    VABDU          v5, v0, v1, v9
+
+    vle8.v         v2, (t2)
+    vle8.v         v3, (t3)
+    add            t2, t2, a7
+    add            t3, t3, a7
+    VABDU          v6, v0, v2, v10
+    VABDU          v7, v0, v3, v11
+.if \x == 4
+    vle8.v         v4, (t4)
+    add            t4, t4, a7
+    VABDU          v8, v0, v4, v12
+.endif
+
+    vsetvli        zero, a7, e32, m\()\lmul2, tu, ma
+
+    vzext.vf4      v12, v5
+    vadd.vv        v16, v16, v12
+
+    vzext.vf4      v0, v6
+    vadd.vv        v20, v20, v0
+
+    vzext.vf4      v4, v7
+    vadd.vv        v24, v24, v4
+.if \x == 4
+    vzext.vf4      v12, v8
+    vadd.vv        v28, v28, v12
+.endif
+
+    sub             t5, t5, a7
+    bgtz            t5, 2b
+
+    // update row pointer
+    addi            a0, a0, FENC_STRIDE
+    add             a1, a1, a5
+    add             a2, a2, a5
+    add             a3, a3, a5
+.if \x == 4
+    add             a4, a4, a5
+.endif
+
+    li              t5, \w          // reset the inline element counter
+    bgtz            t6, 1b
+
+    // reduce and store the result
+    li             t5, \w
+    vsetvli        zero, t5, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v0, 0
+    vredsum.vs     v0, v16, v0
+    vmv.x.s        t0, v0
+    sw             t0, 0(a6)      // scores0
+
+    vmv.v.i        v4, 0
+    vredsum.vs     v4, v20, v4
+    vmv.x.s        t1, v4
+    sw             t1, 4(a6)      // scores1
+
+    vmv.v.i        v8, 0
+    vredsum.vs     v8, v24, v8
+    vmv.x.s        t5, v8
+    sw             t5, 8(a6)      // scores2
+
+.if \x == 4
+    vmv.v.i       v12, 0
+    vredsum.vs    v12, v28, v12
+    vmv.x.s       t6, v12
+
+    sw            t6, 12(a6)      // scores3
+.endif
+    ret
+endfunc
+.endm
+
+SAD_X_FUNC_COMMON 3, 24, 32, 1, 4
+
+SAD_X_FUNC_COMMON 3, 32, 8, 1, 4
+SAD_X_FUNC_COMMON 3, 32, 16, 1, 4
+SAD_X_FUNC_COMMON 3, 32, 24, 1, 4
+SAD_X_FUNC_COMMON 3, 32, 32, 1, 4
+SAD_X_FUNC_COMMON 3, 32, 64, 1, 4
+
+SAD_X_FUNC_COMMON 3, 48, 64, 1, 4
+
+SAD_X_FUNC_COMMON 3, 64, 16, 1, 4
+SAD_X_FUNC_COMMON 3, 64, 32, 1, 4
+SAD_X_FUNC_COMMON 3, 64, 48, 1, 4
+SAD_X_FUNC_COMMON 3, 64, 64, 1, 4
+
+SAD_X_FUNC_COMMON 4, 24, 32, 1, 4
+
+SAD_X_FUNC_COMMON 4, 32, 8, 1, 4
+SAD_X_FUNC_COMMON 4, 32, 16, 1, 4
+SAD_X_FUNC_COMMON 4, 32, 24, 1, 4
+SAD_X_FUNC_COMMON 4, 32, 32, 1, 4
+SAD_X_FUNC_COMMON 4, 32, 64, 1, 4
+
+SAD_X_FUNC_COMMON 4, 48, 64, 1, 4
+
+SAD_X_FUNC_COMMON 4, 64, 16, 1, 4
+SAD_X_FUNC_COMMON 4, 64, 32, 1, 4
+SAD_X_FUNC_COMMON 4, 64, 48, 1, 4
+SAD_X_FUNC_COMMON 4, 64, 64, 1, 4
+
+#else  // HIGH_BIT_DEPTH
+
+.macro SAD_FUNC_SMALL w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li            t0, \h
+    li            t1, \w
+
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v28, 0
+
+    STRIDE_SHIFT  t2, t3, a1, a3, 1, 1
+
+1:
+    addi          t0, t0, -4
+    vsetvli       zero, t1, e16, m\()\lmul1, ta, ma
+
+    vle16.v       v0, (a0)
+    vle16.v       v1, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v28, v0, v1, v16, v17
+
+    vle16.v       v2, (a0)
+    vle16.v       v3, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v28, v2, v3, v18, v19
+
+    vle16.v       v4, (a0)
+    vle16.v       v5, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v28, v4, v5, v20, v21
+
+    vle16.v       v6, (a0)
+    vle16.v       v7, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v28, v6, v7, v22, v23
+
+    bnez          t0, 1b
+
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v24, 0
+    vredsum.vs    v24, v28, v24
+    vmv.x.s       a0, v24
+    ret
+endfunc
+.endm
+
+SAD_FUNC_SMALL 4, 4, 1, 1
+SAD_FUNC_SMALL 4, 8, 1, 1
+SAD_FUNC_SMALL 4, 16, 1, 1
+
+SAD_FUNC_SMALL 8, 4, 1, 2
+SAD_FUNC_SMALL 8, 8, 1, 2
+SAD_FUNC_SMALL 8, 16, 1, 2
+SAD_FUNC_SMALL 8, 32, 1, 2
+
+.macro SAD_FUNC_MEDIUM w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li           t0, \h
+    li           t1, \w
+
+    vsetvli      zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i      v24, 0
+
+    STRIDE_SHIFT t2, t3, a1, a3, 1, 1
+
+1:
+    addi          t0, t0, -2
+    vsetvli       zero, t1, e16, m\()\lmul1, ta, ma
+
+    vle16.v       v0, (a0)
+    vle16.v       v2, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v24, v0, v2, v12, v14
+
+    vle16.v       v4, (a0)
+    vle16.v       v6, (a2)
+    add           a0, a0, t2
+    add           a2, a2, t3
+    UWADBDACCU    v24, v4, v6, v18, v20
+
+    bnez          t0, 1b
+
+    vsetvli       zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v0, 0
+    vredsum.vs    v0, v24, v0
+    vmv.x.s       a0, v0
+    ret
+endfunc
+.endm
+
+SAD_FUNC_MEDIUM 12, 16, 2, 8
+
+SAD_FUNC_MEDIUM 16, 4, 2, 8
+SAD_FUNC_MEDIUM 16, 8, 2, 8
+SAD_FUNC_MEDIUM 16, 12, 2, 8
+SAD_FUNC_MEDIUM 16, 16, 2, 8
+SAD_FUNC_MEDIUM 16, 32, 2, 8
+SAD_FUNC_MEDIUM 16, 64, 2, 8
+
+.macro SAD_FUNC_COMMON w, h, lmul1, lmul2
+function PFX(pixel_sad_\w\()x\h\()_rvv)
+    li             t0, \h
+    li             t1, \w
+
+    vsetvli        zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v24, 0
+
+1:  // row loop
+    mv             t3, a0
+    mv             t4, a2
+    addi           t0, t0, -1
+
+2:  // inline segment loop
+    vsetvli         t5, t1, e16, m\()\lmul1, ta, ma
+
+    vle16.v         v0, (t3)
+    vle16.v         v4, (t4)
+    ADDPTR          a6, a7, t3, t4, t5, t5, 1, 1
+    UWABDU          v16, v0, v4, v8, v12
+
+    vsetvli         zero, t5, e32, m\()\lmul2, tu, ma
+    vadd.vv         v24, v24, v16
+
+    sub             t1, t1, t5
+    bgtz            t1, 2b
+
+    // update row pointer
+    ADDPTR          a6, a7, a0, a2, a1, a3, 1, 1
+
+    li              t1, \w
+    bgtz            t0, 1b
+
+    li              t1, \w
+    vsetvli         zero, t1, e32, m\()\lmul2, ta, ma
+    vmv.v.i         v0, 0
+    vredsum.vs      v0, v24, v0
+    vmv.x.s         a0, v0
+    ret
+endfunc
+.endm
+
+SAD_FUNC_COMMON  24, 32, 1 , 2
+
+SAD_FUNC_COMMON  32, 8, 1 , 2
+SAD_FUNC_COMMON  32, 16, 1 , 2
+SAD_FUNC_COMMON  32, 24, 1 , 2
+SAD_FUNC_COMMON  32, 32, 1 , 2
+SAD_FUNC_COMMON  32, 64, 1 , 2
+
+SAD_FUNC_COMMON  48, 64, 1 , 2
+
+SAD_FUNC_COMMON  64, 16, 1 , 2
+SAD_FUNC_COMMON  64, 32, 1 , 2
+SAD_FUNC_COMMON  64, 48, 1 , 2
+SAD_FUNC_COMMON  64, 64, 1 , 2
+
+//============= SAD_X3 and SAD_X4 code start========================
+// static void x264_pixel_sad_x3_##size(pixel *fenc, pixel *pix0, pixel *pix1, pixel *pix2, intptr_t i_stride, int scores3)
+// static void x264_pixel_sad_x4_##size(pixel *fenc, pixel *pix0, pixel *pix1,pixel *pix2, pixel *pix3, intptr_t i_stride, int scores4)
+.macro SAD_X_FUNC_SMALL x, w, h, lmul1, lmul2
+function PFX(sad_x\x\()_\w\()x\h\()_rvv)
+// Make function arguments for x == 3 look like x == 4.
+.if \x == 3
+    mv             t0, a4
+    mv             t1, a5
+    mv             a5, t0                // a5 = i_stride
+    mv             a6, t1                // a6 = scores
+.endif
+
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add            a5, a5, a5
+    li             t2, FDNC_STRIDE
+
+    li             t3, \w
+    li             t4, \h
+
+    vsetvli        zero, t3, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v24, 0
+    vmv.v.i        v26, 0
+    vmv.v.i        v28, 0
+.if \x == 4
+    vmv.v.i        v30, 0
+.endif
+
+1:
+    addi           t4, t4, -1
+    vsetvli        zero, t3, e16, m\()\lmul1, ta, ma
+
+    vle16.v        v0, (a0)
+    vle16.v        v1, (a1)
+    add            a0, a0, t2
+    add            a1, a1, a5
+    UWADBDACCU     v24, v0, v1, v14, v15
+
+    vle16.v        v2, (a2)
+    vle16.v        v3, (a3)
+    add            a2, a2, a5
+    add            a3, a3, a5
+    UWADBDACCU     v26, v0, v2, v16, v17
+    UWADBDACCU     v28, v0, v3, v18, v19
+.if \x == 4
+    vle16.v        v4, (a4)
+    add            a4, a4, a5
+    UWADBDACCU     v30, v0, v4, v20, v21
+.endif
+
+    bnez           t4, 1b
+
+    // reduce and store the result
+    vsetvli        zero, t3, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v8, 0
+    vredsum.vs     v8, v24, v8
+    vmv.x.s        t0, v8
+    sw             t0, 0(a6)      // scores0
+
+    vmv.v.i        v12, 0
+    vredsum.vs     v12, v26, v12
+    vmv.x.s        t1, v12
+    sw             t1, 4(a6)      // scores1
+
+    vmv.v.i        v4, 0
+    vredsum.vs     v4, v28, v4
+    vmv.x.s        t5, v4
+    sw             t5, 8(a6)      // scores2
+
+.if \x == 4
+    vmv.v.i       v0, 0
+    vredsum.vs    v0, v30, v0
+    vmv.x.s       t6, v0
+
+    sw            t6, 12(a6)      // scores3
+.endif
+    ret
+endfunc
+.endm
+
+SAD_X_FUNC_SMALL 3, 4, 4, 1, 1
+SAD_X_FUNC_SMALL 3, 4, 8, 1, 1
+SAD_X_FUNC_SMALL 3, 4, 16, 1, 1
+
+SAD_X_FUNC_SMALL 3, 8, 4, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 8, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 16, 1, 2
+SAD_X_FUNC_SMALL 3, 8, 32, 1, 2
+
+SAD_X_FUNC_SMALL 4, 4, 4, 1, 1
+SAD_X_FUNC_SMALL 4, 4, 8, 1, 1
+SAD_X_FUNC_SMALL 4, 4, 16, 1, 1
+SAD_X_FUNC_SMALL 4, 8, 4, 1, 2
+
+SAD_X_FUNC_SMALL 4, 8, 8, 1, 2
+SAD_X_FUNC_SMALL 4, 8, 16, 1, 2
+SAD_X_FUNC_SMALL 4, 8, 32, 1, 2
+
+.macro SAD_X_FUNC_COMMON, x, w, h, lmul1, lmul2
+function PFX(sad_x\x\()_\w\()x\h\()_rvv)
+    push
+
+// Make function arguments for x == 3 look like x == 4.
+.if \x == 3
+    mv             t0, a4
+    mv             t1, a5
+    mv             a5, t0
+    mv             a6, t1
+.endif
+
+    // Stride is given in terms of pixel channel size, so double to get number of bytes.
+    add            a5, a5, a5
+
+    li             t5, \w
+    li             t6, \h
+
+    vsetvli       zero, t5, e32, m\()\lmul2, ta, ma
+    vmv.v.i       v22, 0
+    vmv.v.i       v24, 0
+    vmv.v.i       v26, 0
+.if \x == 4
+    vmv.v.i       v28, 0
+.endif
+
+1:  // row loop
+    mv             t0, a0   // current fenc
+    mv             t1, a1   // current pix0
+    mv             t2, a2   // current pix1
+    mv             t3, a3   // current pix2
+.if \x == 4
+    mv             t4, a4   // current pix3
+.endif
+    addi           t6, t6, -1
+
+2:  // inline segment loop
+    vsetvli        a7, t5, e16, m\()\lmul1, ta, ma
+    slli           x9, a7, 1
+
+    vle16.v        v0, (t0)
+    vle16.v        v1, (t1)
+    add            t0, t0, x9
+    add            t1, t1, x9
+    UWABDU         v6, v0, v1, v14, v16
+
+    vle16.v        v2, (t2)
+    vle16.v        v3, (t3)
+    add            t2, t2, x9
+    add            t3, t3, x9
+    UWABDU         v8, v0, v2, v18, v20
+    UWABDU         v10, v0, v3, v14, v16
+.if \x == 4
+    vle16.v        v4, (t4)
+    add            t4, t4, x9
+    UWABDU         v12, v0, v4, v18, v20
+.endif
+
+    vsetvli        zero, a7, e32,  m\()\lmul2, tu, ma
+
+    vadd.vv        v22, v22, v6
+    vadd.vv        v24, v24, v8
+    vadd.vv        v26, v26, v10
+.if \x == 4
+    vadd.vv        v28, v28, v12
+.endif
+
+    sub             t5, t5, a7
+    bgtz            t5, 2b
+
+    // update row pointer
+    addi            a0, a0, FDNC_STRIDE
+    add             a1, a1, a5
+    add             a2, a2, a5
+    add             a3, a3, a5
+.if \x == 4
+    add             a4, a4, a5
+.endif
+
+    li              t5, \w          // reset the inline element counter
+    bgtz            t6, 1b
+
+    // reduce and store the result
+    li             t5, \w
+    vsetvli        zero, t5, e32, m\()\lmul2, ta, ma
+    vmv.v.i        v0, 0
+    vredsum.vs     v0, v22, v0
+    vmv.x.s        t0, v0
+    sw             t0, 0(a6)      // scores0
+
+    vmv.v.i        v4, 0
+    vredsum.vs     v4, v24, v4
+    vmv.x.s        t1, v4
+    sw             t1, 4(a6)      // scores1
+
+    vmv.v.i        v8, 0
+    vredsum.vs     v8, v26, v8
+    vmv.x.s        t5, v8
+    sw             t5, 8(a6)      // scores2
+
+.if \x == 4
+    vmv.v.i       v12, 0
+    vredsum.vs    v12, v28, v12
+    vmv.x.s       t6, v12
+
+    sw            t6, 12(a6)      // scores3
+.endif
+    pop
+    ret
+endfunc
+.endm
+
+SAD_X_FUNC_COMMON 3, 12, 16, 1, 2
+
+SAD_X_FUNC_COMMON 3, 16, 4, 1, 2
+SAD_X_FUNC_COMMON 3, 16, 8, 1, 2
+SAD_X_FUNC_COMMON 3, 16, 12, 1, 2
+SAD_X_FUNC_COMMON 3, 16, 16, 1, 2
+SAD_X_FUNC_COMMON 3, 16, 32, 1, 2
+SAD_X_FUNC_COMMON 3, 16, 64, 1, 2
+
+SAD_X_FUNC_COMMON 3, 24, 32, 1, 2
+
+SAD_X_FUNC_COMMON 3, 32, 8, 1, 2
+SAD_X_FUNC_COMMON 3, 32, 16, 1, 2
+SAD_X_FUNC_COMMON 3, 32, 24, 1, 2
+SAD_X_FUNC_COMMON 3, 32, 32, 1, 2
+SAD_X_FUNC_COMMON 3, 32, 64, 1, 2
+
+SAD_X_FUNC_COMMON 3, 48, 64, 1, 2
+
+SAD_X_FUNC_COMMON 3, 64, 16, 1, 2
+SAD_X_FUNC_COMMON 3, 64, 32, 1, 2
+SAD_X_FUNC_COMMON 3, 64, 48, 1, 2
+SAD_X_FUNC_COMMON 3, 64, 64, 1, 2
+
+
+SAD_X_FUNC_COMMON 4, 12, 16, 1, 2
+
+SAD_X_FUNC_COMMON 4, 16, 4, 1, 2
+SAD_X_FUNC_COMMON 4, 16, 8, 1, 2
+SAD_X_FUNC_COMMON 4, 16, 12, 1, 2
+SAD_X_FUNC_COMMON 4, 16, 16, 1, 2
+SAD_X_FUNC_COMMON 4, 16, 32, 1, 2
+SAD_X_FUNC_COMMON 4, 16, 64, 1, 2
+
+SAD_X_FUNC_COMMON 4, 24, 32, 1, 2
+
+SAD_X_FUNC_COMMON 4, 32, 8, 1, 2
+SAD_X_FUNC_COMMON 4, 32, 16, 1, 2
+SAD_X_FUNC_COMMON 4, 32, 24, 1, 2
+SAD_X_FUNC_COMMON 4, 32, 32, 1, 2
+SAD_X_FUNC_COMMON 4, 32, 64, 1, 2
+
+SAD_X_FUNC_COMMON 4, 48, 64, 1, 2
+
+SAD_X_FUNC_COMMON 4, 64, 16, 1, 2
+SAD_X_FUNC_COMMON 4, 64, 32, 1, 2
+SAD_X_FUNC_COMMON 4, 64, 48, 1, 2
+SAD_X_FUNC_COMMON 4, 64, 64, 1, 2
+
+#endif // !HIGH_BIT_DEPTH
\ No newline at end of file

x265-4.2.tar/source/common/riscv64/sao-prim.cpp Added

@@ -0,0 +1,229 @@
+/*****************************************************************************
+ * Copyright (C) 2024 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include <riscv_vector.h>
+#include <stdint.h>
+#include "sao.h"
+#include "primitives.h"
+
+static inline void compute_eo_stats(const vint8m1_t edge_type, const int16_t *diff,
+    int32_t *stats, int32_t *count, int vl)
+{
+    (void)__riscv_vsetvl_e16m2(vl);
+    vint16m2_t tmp_0 = __riscv_vmv_v_x_i16m2(0, vl);
+    (void)__riscv_vsetvl_e32m1(vl);
+    vint32m1_t tmp_01 = __riscv_vmv_v_x_i32m1(0, vl);
+
+    // Create a mask for each edge type.
+    (void)__riscv_vsetvl_e8m1(vl);
+    vbool8_t mask0 = __riscv_vmseq_vx_i8m1_b8(edge_type, -2, vl);
+    vbool8_t mask1 = __riscv_vmseq_vx_i8m1_b8(edge_type, -1, vl);
+    vbool8_t mask2 = __riscv_vmseq_vx_i8m1_b8(edge_type, 0, vl);
+    vbool8_t mask3 = __riscv_vmseq_vx_i8m1_b8(edge_type, 1, vl);
+    vbool8_t mask4 = __riscv_vmseq_vx_i8m1_b8(edge_type, 2, vl);
+
+    count1 += __riscv_vcpop_m_b8(mask0, vl);
+    count2 += __riscv_vcpop_m_b8(mask1, vl);
+    count0 += __riscv_vcpop_m_b8(mask2, vl);
+    count3 += __riscv_vcpop_m_b8(mask3, vl);
+    count4 += __riscv_vcpop_m_b8(mask4, vl);
+
+    // Widen the masks to 16-bit.
+    (void)__riscv_vsetvl_e16m2(vl);
+    vint16m2_t load_diff = __riscv_vle16_v_i16m2(diff, vl);
+    vint16m2_t temp_add0 = __riscv_vmerge_vvm_i16m2(tmp_0, load_diff, mask0, vl);
+    vint16m2_t temp_add1 = __riscv_vmerge_vvm_i16m2(tmp_0, load_diff, mask1, vl);
+    vint16m2_t temp_add2 = __riscv_vmerge_vvm_i16m2(tmp_0, load_diff, mask2, vl);
+    vint16m2_t temp_add3 = __riscv_vmerge_vvm_i16m2(tmp_0, load_diff, mask3, vl);
+    vint16m2_t temp_add4 = __riscv_vmerge_vvm_i16m2(tmp_0, load_diff, mask4, vl);
+
+    vint32m1_t temp_stats0 = __riscv_vwredsum_vs_i16m2_i32m1(temp_add0, tmp_01, vl);
+    vint32m1_t temp_stats1 = __riscv_vwredsum_vs_i16m2_i32m1(temp_add1, tmp_01, vl);
+    vint32m1_t temp_stats2 = __riscv_vwredsum_vs_i16m2_i32m1(temp_add2, tmp_01, vl);
+    vint32m1_t temp_stats3 = __riscv_vwredsum_vs_i16m2_i32m1(temp_add3, tmp_01, vl);
+    vint32m1_t temp_stats4 = __riscv_vwredsum_vs_i16m2_i32m1(temp_add4, tmp_01, vl);
+
+    stats1 += __riscv_vmv_x_s_i32m1_i32(temp_stats0);
+    stats2 += __riscv_vmv_x_s_i32m1_i32(temp_stats1);
+    stats0 += __riscv_vmv_x_s_i32m1_i32(temp_stats2);
+    stats3 += __riscv_vmv_x_s_i32m1_i32(temp_stats3);
+    stats4 += __riscv_vmv_x_s_i32m1_i32(temp_stats4);
+}
+
+static inline vint8m1_t signOf_rvv(const pixel *a, const pixel *b, int vl)
+{
+#if HIGH_BIT_DEPTH
+    vl = __riscv_vsetvl_e16m2(vl);
+    vuint16m2_t s0 = __riscv_vle16_v_u16m2(a, vl);
+    vuint16m2_t s1 = __riscv_vle16_v_u16m2(b, vl);
+
+    vbool8_t bgt = __riscv_vmsgtu_vv_u16m2_b8(s0, s1, vl);
+    vbool8_t blt = __riscv_vmsltu_vv_u16m2_b8(s0, s1, vl);
+
+    // a > b : 1   a == b : 0  a < b : -1
+    vl = __riscv_vsetvl_e8m1(vl);
+    vint8m1_t ret = __riscv_vmv_v_x_i8m1(0, vl);
+    ret = __riscv_vmerge_vxm_i8m1(ret, 1, bgt, vl);
+    ret = __riscv_vmerge_vxm_i8m1(ret, -1, blt, vl);
+#else // HIGH_BIT_DEPTH
+    vl = __riscv_vsetvl_e8m1(vl);
+    vint8m1_t ret = __riscv_vmv_v_x_i8m1(0, vl);
+
+    vuint8m1_t s0 = __riscv_vle8_v_u8m1(a, vl);
+    vuint8m1_t s1 = __riscv_vle8_v_u8m1(b, vl);
+
+    // a > b : 1   a == b : 0  a < b : -1
+    vbool8_t bgt = __riscv_vmsgtu_vv_u8m1_b8(s0, s1, vl);
+    ret = __riscv_vmerge_vxm_i8m1(ret, 1, bgt, vl);
+    vbool8_t blt = __riscv_vmsltu_vv_u8m1_b8(s0, s1, vl);
+    ret = __riscv_vmerge_vxm_i8m1(ret, -1, blt, vl);
+#endif // HIGH_BIT_DEPTH
+
+    return ret;
+}
+
+namespace X265_NS {
+void saoCuStatsBO_rvv(const int16_t *diff, const pixel *rec, intptr_t stride, int endX, int endY, int32_t *stats, int32_t *count)
+{
+#if HIGH_BIT_DEPTH
+    const int n_elem = 4;
+    const int elem_width = 16;
+#else
+    const int n_elem = 8;
+    const int elem_width = 8;
+#endif
+
+    // Additional temporary buffer for accumulation.
+    int32_t stats_tmp32 = { 0 };
+    int32_t count_tmp32 = { 0 };
+
+    // Byte-addressable pointers to buffers, to optimise address calculation.
+    uint8_t *stats_b2 = {
+        reinterpret_cast<uint8_t *>(stats),
+        reinterpret_cast<uint8_t *>(stats_tmp),
+    };
+    uint8_t *count_b2 = {
+        reinterpret_cast<uint8_t *>(count),
+        reinterpret_cast<uint8_t *>(count_tmp),
+    };
+
+    // Combine shift for index calculation with shift for address calculation.
+    const int right_shift = X265_DEPTH - X265_NS::SAO::SAO_BO_BITS;
+    const int left_shift = 2;
+    const int shift = right_shift - left_shift;
+    // Mask out bits 7, 1 & 0 to account for combination of shifts.
+    const int mask = 0x7c;
+
+    // Compute statistics into temporary buffers.
+    for (int y = 0; y < endY; y++)
+    {
+        int x = 0;
+        for (; x + n_elem < endX; x += n_elem)
+        {
+            uint64_t class_idx_64 =
+                *reinterpret_cast<const uint64_t *>(rec + x) >> shift;
+
+            for (int i = 0; i < n_elem; ++i)
+            {
+                const int idx = i & 1;
+                const int off  = (class_idx_64 >> (i * elem_width)) & mask;
+                *reinterpret_cast<uint32_t*>(stats_bidx + off) += diffx + i;
+                *reinterpret_cast<uint32_t*>(count_bidx + off) += 1;
+            }
+        }
+
+        if (x < endX)
+        {
+            uint64_t class_idx_64 =
+                *reinterpret_cast<const uint64_t *>(rec + x) >> shift;
+
+            for (int i = 0; (i + x) < endX; ++i)
+            {
+                const int idx = i & 1;
+                const int off  = (class_idx_64 >> (i * elem_width)) & mask;
+                *reinterpret_cast<uint32_t*>(stats_bidx + off) += diffx + i;
+                *reinterpret_cast<uint32_t*>(count_bidx + off) += 1;
+            }
+        }
+
+        diff += MAX_CU_SIZE;
+        rec += stride;
+    }
+
+    // Reduce temporary buffers to destination using Neon.
+    for (int i = 0; i < 32;)
+    {
+        int vl = __riscv_vsetvl_e32m2(32 - i);
+        vint32m2_t s0 = __riscv_vle32_v_i32m2(stats_tmp + i, vl);
+        vint32m2_t s1 = __riscv_vle32_v_i32m2(stats + i, vl);
+        vint32m2_t ss = __riscv_vadd_vv_i32m2(s0, s1, vl);
+        __riscv_vse32_v_i32m2(stats + i, ss, vl);
+
+        vint32m2_t c0 = __riscv_vle32_v_i32m2(count_tmp + i, vl);
+        vint32m2_t c1 = __riscv_vle32_v_i32m2(count + i, vl);
+        vint32m2_t cs = __riscv_vadd_vv_i32m2(c0, c1, vl);
+        __riscv_vse32_v_i32m2(count + i, cs, vl);
+
+        i += vl;
+    }
+}
+
+void saoCuStatsE0_rvv(const int16_t *diff, const pixel *rec, intptr_t stride, int endX, int endY, int32_t *stats, int32_t *count)
+{
+    for (int y = 0; y < endY; y++)
+    {
+        // Calculate negated sign_left(x) directly, to save negation when
+        // reusing sign_right(x) as sign_left(x + 1).
+        int vl = __riscv_vsetvl_e8m1(endX);
+        vint8m1_t neg_sign_left = __riscv_vmv_v_x_i8m1(x265_signOf(rec-1 - rec0), vl);
+        for (int x = 0; x < endX; x += vl)
+        {
+            vl = __riscv_vsetvl_e8m1(endX - x);
+            vint8m1_t sign_right = signOf_rvv(rec + x, rec + x + 1, vl);
+
+            // neg_sign_left(x) = sign_right(x + 1), reusing one from previous
+            // iteration.
+            neg_sign_left = __riscv_vslideup_vx_i8m1(neg_sign_left, sign_right, 1, vl);
+
+            // Subtract instead of add, as sign_left is negated.
+            vint8m1_t edge_type = __riscv_vsub_vv_i8m1(sign_right, neg_sign_left, vl);
+
+            // For reuse in the next iteration.
+            neg_sign_left = __riscv_vslidedown_vx_i8m1(sign_right, vl - 1, vl);
+
+            compute_eo_stats(edge_type, diff + x, stats, count, vl);
+        }
+
+        diff += MAX_CU_SIZE;
+        rec += stride;
+    }
+}
+
+
+void setupSaoPrimitives_rvv(EncoderPrimitives &p)
+{
+    p.saoCuStatsBO = saoCuStatsBO_rvv;
+    p.saoCuStatsE0 = saoCuStatsE0_rvv;
+}
+
+}// namespace X265_NS

x265-4.2.tar/source/common/riscv64/sao.S Added

@@ -0,0 +1,884 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Changsheng Wu <wu.changsheng@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+.text
+
+//void saoCuStatsE0_c(const int16_t *diff, const pixel *rec, intptr_t stride, int endX, int endY, int32_t *stats, int32_t *count)
+function PFX(saoCuStatsE0_rvv)
+    addi            sp, sp, -64
+    sd              s0, (sp)
+    sd              s1, 8(sp)
+    sd              s2, 16(sp)
+    sd              s3, 24(sp)
+    sd              s4, 32(sp)
+    sd              s5, 40(sp)
+    sd              s6, 48(sp)
+    li              s0, 0
+    li              s1, 0
+    li              s2, 0
+    li              s3, 0
+    li              s4, 0
+    li              s6, 1 << 7
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+    vsetvli         t1, zero, e32, m2, ta, ma
+#else
+    vsetvli         t1, zero, e32, m4, ta, ma
+#endif
+    vmv.v.i         v4, 0
+    vmv.v.i         v8, 0
+    vmv.v.i         v12, 0
+    vmv.v.i         v16, 0
+    vmv.v.i         v20, 0
+#if HIGH_BIT_DEPTH
+    vsetvli         t1, zero, e16, m1, ta, ma
+#else
+    vsetvli         t1, zero, e16, m2, ta, ma
+#endif
+    vmv.v.i         v30, 0
+
+loop_saoCuStatsE0_y:
+    mv              t0, a0
+    mv              t1, a1
+    mv              t2, a3
+#if HIGH_BIT_DEPTH
+    lhu             t3, -2(a1)
+    lhu             t4, (a1)
+#else
+    lbu             t3, -1(a1)
+    lbu             t4, (a1)
+#endif
+    sub             t3, t3, t4
+    SIGNOF          t3, t4, t5
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, a3, e16, m1, ta, ma
+#else
+    vsetvli         zero, a3, e8, m1, ta, ma
+#endif
+    vmv.s.x         v1, t3
+loop_saoCuStatsE0_x:
+#if HIGH_BIT_DEPTH
+    addi            t5, t1, 2
+    vsetvli         t3, t2, e16, m1, ta, ma
+    vle16.v         v2, (t1)
+    vle16.v         v3, (t5)
+#else
+    addi            t5, t1, 1
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v2, (t1)
+    vle8.v          v3, (t5)
+#endif
+    addi            t4, t3, -1
+    SIGNOF_RVV      v24, v2, v3
+    vslideup.vi     v1, v24, 1
+    vsub.vv         v25, v24, v1
+    vslidedown.vx   v1, v24, t4
+
+    vmseq.vi        v0, v25, -2
+    vmseq.vi        v2, v25, -1
+    vmseq.vi        v3, v25, 0
+    vmseq.vi        v24, v25, 1
+    vmseq.vi        v25, v25, 2
+
+    vcpop.m         t4, v0
+    vcpop.m         t5, v2
+    vcpop.m         t6, v3
+    vcpop.m         a7, v24
+    vcpop.m         s5, v25
+    add             s0, s0, t4
+    add             s1, s1, t5
+    add             s2, s2, t6
+    add             s3, s3, a7
+    add             s4, s4, s5
+
+#if HIGH_BIT_DEPTH
+    vle16.v         v26, (t0)
+    vmerge.vvm      v6, v30, v26, v0
+    vmv1r.v         v0, v2
+    vmerge.vvm      v10, v30, v26, v0
+    vmv1r.v         v0, v3
+    vmerge.vvm      v14, v30, v26, v0
+    vmv1r.v         v0, v24
+    vmerge.vvm      v18, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v22, v30, v26, v0
+    vsetvli         zero, t3, e16, m1, tu, ma
+    vwadd.wv        v4, v4, v6
+    vwadd.wv        v8, v8, v10
+    vwadd.wv        v12, v12, v14
+    vwadd.wv        v16, v16, v18
+    vwadd.wv        v20, v20, v22
+#else
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vle16.v         v26, (t0)
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v4, v4, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v2
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v8, v8, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v3
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v12, v12, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v24
+    vmerge.vvm      v28, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v2, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v16, v16, v28
+    vwadd.wv        v20, v20, v2
+#endif
+
+    slli            t4, t3, 1
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t4
+#else
+    add             t1, t1, t3
+#endif
+    sub             t2, t2, t3
+    add             t0, t0, t4
+    bgtz            t2, loop_saoCuStatsE0_x
+    add             a0, a0, s6
+    add             a1, a1, a2
+    addi            a4, a4, -1
+    bgtz            a4, loop_saoCuStatsE0_y
+
+    lw              t0, (a6)
+    lw              t1, 4(a6)
+    lw              t2, 8(a6)
+    lw              t3, 12(a6)
+    lw              t4, 16(a6)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a6)
+    sw              t1, 4(a6)
+    sw              t2, 8(a6)
+    sw              t3, 12(a6)
+    sw              t4, 16(a6)
+
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, a3, e32, m2, ta, ma
+#else
+    vsetvli         zero, a3, e32, m4, ta, ma
+#endif
+    vredsum.vs      v24, v4, v30
+    vredsum.vs      v25, v8, v30
+    vredsum.vs      v26, v12, v30
+    vredsum.vs      v27, v16, v30
+    vredsum.vs      v28, v20, v30
+    vsetvli         zero, a3, e32, m1, ta, ma
+    vmv.x.s         s0, v24
+    vmv.x.s         s1, v25
+    vmv.x.s         s2, v26
+    vmv.x.s         s3, v27
+    vmv.x.s         s4, v28
+    lw              t0, (a5)
+    lw              t1, 4(a5)
+    lw              t2, 8(a5)
+    lw              t3, 12(a5)
+    lw              t4, 16(a5)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a5)
+    sw              t1, 4(a5)
+    sw              t2, 8(a5)
+    sw              t3, 12(a5)
+    sw              t4, 16(a5)
+
+    ld              s0, (sp)
+    ld              s1, 8(sp)
+    ld              s2, 16(sp)
+    ld              s3, 24(sp)
+    ld              s4, 32(sp)
+    ld              s5, 40(sp)
+    ld              s6, 48(sp)
+    addi            sp, sp, 64
+    ret
+endfunc
+
+//void saoCuStatsE1_c(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int endX, int endY, int32_t *stats, int32_t *count)
+function PFX(saoCuStatsE1_rvv)
+    addi            sp, sp, -64
+    sd              s0, (sp)
+    sd              s1, 8(sp)
+    sd              s2, 16(sp)
+    sd              s3, 24(sp)
+    sd              s4, 32(sp)
+    sd              s5, 40(sp)
+    sd              s6, 48(sp)
+    sd              s7, 56(sp)
+    li              s0, 0
+    li              s1, 0
+    li              s2, 0
+    li              s3, 0
+    li              s4, 0
+    li              s6, 1 << 7
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+    vsetvli         t1, zero, e32, m2, ta, ma
+#else
+    vsetvli         t1, zero, e32, m4, ta, ma
+#endif
+    vmv.v.i         v4, 0
+    vmv.v.i         v8, 0
+    vmv.v.i         v12, 0
+    vmv.v.i         v16, 0
+    vmv.v.i         v20, 0
+#if HIGH_BIT_DEPTH
+    vsetvli         t1, zero, e16, m1, ta, ma
+#else
+    vsetvli         t1, zero, e16, m2, ta, ma
+#endif
+    vmv.v.i         v30, 0
+
+loop_saoCuStatsE1_y:
+    mv              t0, a0
+    mv              t1, a1
+    mv              t2, a4
+    mv              t4, a3
+loop_saoCuStatsE1_x:
+    add             t5, t1, a2
+#if HIGH_BIT_DEPTH
+    vsetvli         t3, t2, e16, m1, ta, ma
+    vle16.v         v2, (t1)
+    vle16.v         v3, (t5)
+    vle8.v          v1, (t4)
+    vsext.vf2       v26, v1
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v26
+    vsetvli         zero, t3, e8, m1, ta, ma
+    vnsrl.wi        v26, v24, 0
+    vrsub.vi        v1, v26, 0
+    vse8.v          v1, (t4)
+    vsetvli         zero, t3, e16, m1, ta, ma
+#else
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v2, (t1)
+    vle8.v          v3, (t5)
+    vle8.v          v1, (t4)
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v1
+    vrsub.vi        v1, v24, 0
+    vse8.v          v1, (t4)
+#endif
+
+    vmseq.vi        v0, v25, -2
+    vmseq.vi        v2, v25, -1
+    vmseq.vi        v3, v25, 0
+    vmseq.vi        v24, v25, 1
+    vmseq.vi        v25, v25, 2
+
+    vcpop.m         s7, v0
+    add             s0, s0, s7
+    vcpop.m         t5, v2
+    vcpop.m         t6, v3
+    vcpop.m         s7, v24
+    vcpop.m         s5, v25
+    add             s1, s1, t5
+    add             s2, s2, t6
+    add             s3, s3, s7
+    add             s4, s4, s5
+
+#if HIGH_BIT_DEPTH
+    vle16.v         v26, (t0)
+    vmerge.vvm      v6, v30, v26, v0
+    vmv1r.v         v0, v2
+    vmerge.vvm      v10, v30, v26, v0
+    vmv1r.v         v0, v3
+    vmerge.vvm      v14, v30, v26, v0
+    vmv1r.v         v0, v24
+    vmerge.vvm      v18, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v22, v30, v26, v0
+    vsetvli         zero, t3, e16, m1, tu, ma
+    vwadd.wv        v4, v4, v6
+    vwadd.wv        v8, v8, v10
+    vwadd.wv        v12, v12, v14
+    vwadd.wv        v16, v16, v18
+    vwadd.wv        v20, v20, v22
+#else
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vle16.v         v26, (t0)
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v4, v4, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v2
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v8, v8, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v3
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v12, v12, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v24
+    vmerge.vvm      v28, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v2, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v16, v16, v28
+    vwadd.wv        v20, v20, v2
+#endif
+
+    slli            t5, t3, 1
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t5
+#else
+    add             t1, t1, t3
+#endif
+    add             t4, t4, t3
+    sub             t2, t2, t3
+    add             t0, t0, t5
+    bgtz            t2, loop_saoCuStatsE1_x
+    add             a0, a0, s6
+    add             a1, a1, a2
+    addi            a5, a5, -1
+    bgtz            a5, loop_saoCuStatsE1_y
+
+    lw              t0, (a7)
+    lw              t1, 4(a7)
+    lw              t2, 8(a7)
+    lw              t3, 12(a7)
+    lw              t4, 16(a7)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a7)
+    sw              t1, 4(a7)
+    sw              t2, 8(a7)
+    sw              t3, 12(a7)
+    sw              t4, 16(a7)
+
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, a3, e32, m2, ta, ma
+#else
+    vsetvli         zero, a3, e32, m4, ta, ma
+#endif
+    vredsum.vs      v24, v4, v30
+    vredsum.vs      v25, v8, v30
+    vredsum.vs      v26, v12, v30
+    vredsum.vs      v27, v16, v30
+    vredsum.vs      v28, v20, v30
+    vsetvli         zero, a3, e32, m1, ta, ma
+    vmv.x.s         s0, v24
+    vmv.x.s         s1, v25
+    vmv.x.s         s2, v26
+    vmv.x.s         s3, v27
+    vmv.x.s         s4, v28
+    lw              t0, (a6)
+    lw              t1, 4(a6)
+    lw              t2, 8(a6)
+    lw              t3, 12(a6)
+    lw              t4, 16(a6)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a6)
+    sw              t1, 4(a6)
+    sw              t2, 8(a6)
+    sw              t3, 12(a6)
+    sw              t4, 16(a6)
+
+    ld              s0, (sp)
+    ld              s1, 8(sp)
+    ld              s2, 16(sp)
+    ld              s3, 24(sp)
+    ld              s4, 32(sp)
+    ld              s5, 40(sp)
+    ld              s6, 48(sp)
+    ld              s7, 56(sp)
+    addi            sp, sp, 64
+    ret
+endfunc
+
+//void saoCuStatsE2_c(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int8_t *upBufft, int endX, int endY, int32_t *stats, int32_t *count)
+function PFX(saoCuStatsE2_rvv)
+    ld              t6, (sp)
+    addi            sp, sp, -80
+    sd              s0, (sp)
+    sd              s1, 8(sp)
+    sd              s2, 16(sp)
+    sd              s3, 24(sp)
+    sd              s4, 32(sp)
+    sd              s5, 40(sp)
+    sd              s6, 48(sp)
+    sd              s7, 56(sp)
+    sd              s8, 64(sp)
+    sd              s9, 72(sp)
+    li              s0, 0
+    li              s1, 0
+    li              s2, 0
+    li              s3, 0
+    li              s4, 0
+    li              s6, 1 << 7
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+    addi            s5, a2, 2
+    vsetvli         t1, zero, e32, m2, ta, ma
+#else
+    addi            s5, a2, 1
+    vsetvli         t1, zero, e32, m4, ta, ma
+#endif
+    vmv.v.i         v4, 0
+    vmv.v.i         v8, 0
+    vmv.v.i         v12, 0
+    vmv.v.i         v16, 0
+    vmv.v.i         v20, 0
+#if HIGH_BIT_DEPTH
+    vsetvli         t1, zero, e16, m1, ta, ma
+#else
+    vsetvli         t1, zero, e16, m2, ta, ma
+#endif
+    vmv.v.i         v30, 0
+
+loop_saoCuStatsE2_y:
+    mv              t0, a0
+    mv              t1, a1
+    mv              t2, a5
+    mv              t4, a3
+    mv              t5, a4
+    add             s7, a1, a2
+#if HIGH_BIT_DEPTH
+    lhu             s8, (s7)
+    lhu             s9, -2(a1)
+#else
+    lbu             s8, (s7)
+    lbu             s9, -1(a1)
+#endif
+    sub             s8, s8, s9
+    SIGNOF          s8, s7, s9
+    sb              s8, (a4)
+loop_saoCuStatsE2_x:
+    add             s7, t1, s5
+    addi            s8, t5, 1
+#if HIGH_BIT_DEPTH
+    vsetvli         t3, t2, e16, m1, ta, ma
+    vle16.v         v2, (t1)
+    vle16.v         v3, (s7)
+    vle8.v          v1, (t4)
+    vsext.vf2       v26, v1
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v26
+    vrsub.vi        v24, v24, 0
+    vsetvli         zero, t3, e8, m1, ta, ma
+    vnsrl.wi        v26, v24, 0
+    vse8.v          v26, (s8)
+    vsetvli         zero, t3, e16, m1, ta, ma
+#else
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v2, (t1)
+    vle8.v          v3, (s7)
+    vle8.v          v1, (t4)
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v1
+    vrsub.vi        v24, v24, 0
+    vse8.v          v24, (s8)
+#endif
+
+    vmseq.vi        v0, v25, -2
+    vmseq.vi        v2, v25, -1
+    vmseq.vi        v3, v25, 0
+    vmseq.vi        v24, v25, 1
+    vmseq.vi        v25, v25, 2
+
+    vcpop.m         s7, v0
+    add             s0, s0, s7
+    vcpop.m         s8, v2
+    add             s1, s1, s8
+    vcpop.m         s7, v3
+    vcpop.m         s8, v24
+    vcpop.m         s9, v25
+    add             s2, s2, s7
+    add             s3, s3, s8
+    add             s4, s4, s9
+
+#if HIGH_BIT_DEPTH
+    vle16.v         v26, (t0)
+    vmerge.vvm      v6, v30, v26, v0
+    vmv1r.v         v0, v2
+    vmerge.vvm      v10, v30, v26, v0
+    vmv1r.v         v0, v3
+    vmerge.vvm      v14, v30, v26, v0
+    vmv1r.v         v0, v24
+    vmerge.vvm      v18, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v22, v30, v26, v0
+    vsetvli         zero, t3, e16, m1, tu, ma
+    vwadd.wv        v4, v4, v6
+    vwadd.wv        v8, v8, v10
+    vwadd.wv        v12, v12, v14
+    vwadd.wv        v16, v16, v18
+    vwadd.wv        v20, v20, v22
+#else
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vle16.v         v26, (t0)
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v4, v4, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v2
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v8, v8, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v3
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v12, v12, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v24
+    vmerge.vvm      v28, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v2, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v16, v16, v28
+    vwadd.wv        v20, v20, v2
+#endif
+
+    slli            s7, t3, 1
+#if HIGH_BIT_DEPTH
+    add             t1, t1, s7
+#else
+    add             t1, t1, t3
+#endif
+    add             t4, t4, t3
+    add             t5, t5, t3
+    sub             t2, t2, t3
+    add             t0, t0, s7
+    bgtz            t2, loop_saoCuStatsE2_x
+    mv              s8, a4
+    add             a0, a0, s6
+    add             a1, a1, a2
+    mv              a4, a3
+    addi            a6, a6, -1
+    mv              a3, s8
+    bgtz            a6, loop_saoCuStatsE2_y
+
+    lw              t0, (t6)
+    lw              t1, 4(t6)
+    lw              t2, 8(t6)
+    lw              t3, 12(t6)
+    lw              t4, 16(t6)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (t6)
+    sw              t1, 4(t6)
+    sw              t2, 8(t6)
+    sw              t3, 12(t6)
+    sw              t4, 16(t6)
+
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, a3, e32, m2, ta, ma
+#else
+    vsetvli         zero, a3, e32, m4, ta, ma
+#endif
+    vredsum.vs      v24, v4, v30
+    vredsum.vs      v25, v8, v30
+    vredsum.vs      v26, v12, v30
+    vredsum.vs      v27, v16, v30
+    vredsum.vs      v28, v20, v30
+    vsetvli         zero, a3, e32, m1, ta, ma
+    vmv.x.s         s0, v24
+    vmv.x.s         s1, v25
+    vmv.x.s         s2, v26
+    vmv.x.s         s3, v27
+    vmv.x.s         s4, v28
+    lw              t0, (a7)
+    lw              t1, 4(a7)
+    lw              t2, 8(a7)
+    lw              t3, 12(a7)
+    lw              t4, 16(a7)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a7)
+    sw              t1, 4(a7)
+    sw              t2, 8(a7)
+    sw              t3, 12(a7)
+    sw              t4, 16(a7)
+
+    ld              s0, (sp)
+    ld              s1, 8(sp)
+    ld              s2, 16(sp)
+    ld              s3, 24(sp)
+    ld              s4, 32(sp)
+    ld              s5, 40(sp)
+    ld              s6, 48(sp)
+    ld              s7, 56(sp)
+    ld              s8, 64(sp)
+    ld              s9, 72(sp)
+    addi            sp, sp, 80
+    ret
+endfunc
+
+//void saoCuStatsE3_c(const int16_t *diff, const pixel *rec, intptr_t stride, int8_t *upBuff1, int endX, int endY, int32_t *stats, int32_t *count)
+function PFX(saoCuStatsE3_rvv)
+    addi            sp, sp, -64
+    sd              s0, (sp)
+    sd              s1, 8(sp)
+    sd              s2, 16(sp)
+    sd              s3, 24(sp)
+    sd              s4, 32(sp)
+    sd              s5, 40(sp)
+    sd              s6, 48(sp)
+    sd              s7, 56(sp)
+    li              s0, 0
+    li              s1, 0
+    li              s2, 0
+    li              s3, 0
+    li              s4, 0
+    li              s6, 1 << 7
+#if HIGH_BIT_DEPTH
+    slli            a2, a2, 1
+    addi            s5, a2, -2
+    vsetvli         t1, zero, e32, m2, ta, ma
+#else
+    addi            s5, a2, -1
+    vsetvli         t1, zero, e32, m4, ta, ma
+#endif
+    vmv.v.i         v4, 0
+    vmv.v.i         v8, 0
+    vmv.v.i         v12, 0
+    vmv.v.i         v16, 0
+    vmv.v.i         v20, 0
+#if HIGH_BIT_DEPTH
+    vsetvli         t1, zero, e16, m1, ta, ma
+#else
+    vsetvli         t1, zero, e16, m2, ta, ma
+#endif
+    vmv.v.i         v30, 0
+
+loop_saoCuStatsE3_y:
+    mv              t0, a0
+    mv              t1, a1
+    mv              t2, a4
+    mv              t4, a3
+loop_saoCuStatsE3_x:
+    add             t5, t1, s5
+#if HIGH_BIT_DEPTH
+    vsetvli         t3, t2, e16, m1, ta, ma
+    vle16.v         v2, (t1)
+    vle16.v         v3, (t5)
+    vle8.v          v1, (t4)
+    vsext.vf2       v26, v1
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v26
+    vsetvli         zero, t3, e8, m1, ta, ma
+    vnsrl.wi        v26, v24, 0
+    vrsub.vi        v1, v26, 0
+    addi            t6, t4, -1
+    vse8.v          v1, (t6)
+    vsetvli         zero, t3, e16, m1, ta, ma
+#else
+    vsetvli         t3, t2, e8, m1, ta, ma
+    vle8.v          v2, (t1)
+    vle8.v          v3, (t5)
+    vle8.v          v1, (t4)
+    SIGNOF_RVV      v24, v2, v3
+    vadd.vv         v25, v24, v1
+    vrsub.vi        v1, v24, 0
+    addi            t6, t4, -1
+    vse8.v          v1, (t6)
+#endif
+
+    vmseq.vi        v0, v25, -2
+    vmseq.vi        v2, v25, -1
+    vmseq.vi        v3, v25, 0
+    vmseq.vi        v24, v25, 1
+    vmseq.vi        v25, v25, 2
+
+    vcpop.m         s7, v0
+    add             s0, s0, s7
+    vcpop.m         t5, v2
+    add             s1, s1, t5
+    vcpop.m         t6, v3
+    vcpop.m         s7, v24
+    vcpop.m         t5, v25
+    add             s2, s2, t6
+    add             s3, s3, s7
+    add             s4, s4, t5
+
+#if HIGH_BIT_DEPTH
+    vle16.v         v26, (t0)
+    vmerge.vvm      v6, v30, v26, v0
+    vmv1r.v         v0, v2
+    vmerge.vvm      v10, v30, v26, v0
+    vmv1r.v         v0, v3
+    vmerge.vvm      v14, v30, v26, v0
+    vmv1r.v         v0, v24
+    vmerge.vvm      v18, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v22, v30, v26, v0
+    vsetvli         zero, t3, e16, m1, tu, ma
+    vwadd.wv        v4, v4, v6
+    vwadd.wv        v8, v8, v10
+    vwadd.wv        v12, v12, v14
+    vwadd.wv        v16, v16, v18
+    vwadd.wv        v20, v20, v22
+#else
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vle16.v         v26, (t0)
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v4, v4, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v2
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v8, v8, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v3
+    vmerge.vvm      v28, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v12, v12, v28
+    vsetvli         zero, t3, e16, m2, ta, ma
+    vmv1r.v         v0, v24
+    vmerge.vvm      v28, v30, v26, v0
+    vmv1r.v         v0, v25
+    vmerge.vvm      v2, v30, v26, v0
+    vsetvli         zero, t3, e16, m2, tu, ma
+    vwadd.wv        v16, v16, v28
+    vwadd.wv        v20, v20, v2
+#endif
+
+    slli            t5, t3, 1
+#if HIGH_BIT_DEPTH
+    add             t1, t1, t5
+#else
+    add             t1, t1, t3
+#endif
+    add             t4, t4, t3
+    sub             t2, t2, t3
+    add             t0, t0, t5
+    bgtz            t2, loop_saoCuStatsE3_x
+#if HIGH_BIT_DEPTH
+    slli            t5, a4, 1
+    add             t5, t5, a1
+    lhu             t6, (t5)
+    add             t5, t5, a2
+    lhu             s7, -2(t5)
+#else
+    add             t5, a1, a4
+    lbu             t6, (t5)
+    add             t5, t5, a2
+    lbu             s7, -1(t5)
+#endif
+    sub             t5, s7, t6
+    SIGNOF          t5, s7, t6
+    add             t6, a3, a4
+    sb              t5, -1(t6)
+    add             a0, a0, s6
+    add             a1, a1, a2
+    addi            a5, a5, -1
+    bgtz            a5, loop_saoCuStatsE3_y
+
+    lw              t0, (a7)
+    lw              t1, 4(a7)
+    lw              t2, 8(a7)
+    lw              t3, 12(a7)
+    lw              t4, 16(a7)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a7)
+    sw              t1, 4(a7)
+    sw              t2, 8(a7)
+    sw              t3, 12(a7)
+    sw              t4, 16(a7)
+
+#if HIGH_BIT_DEPTH
+    vsetvli         zero, a3, e32, m2, ta, ma
+#else
+    vsetvli         zero, a3, e32, m4, ta, ma
+#endif
+    vredsum.vs      v24, v4, v30
+    vredsum.vs      v25, v8, v30
+    vredsum.vs      v26, v12, v30
+    vredsum.vs      v27, v16, v30
+    vredsum.vs      v28, v20, v30
+    vsetvli         zero, a3, e32, m1, ta, ma
+    vmv.x.s         s0, v24
+    vmv.x.s         s1, v25
+    vmv.x.s         s2, v26
+    vmv.x.s         s3, v27
+    vmv.x.s         s4, v28
+    lw              t0, (a6)
+    lw              t1, 4(a6)
+    lw              t2, 8(a6)
+    lw              t3, 12(a6)
+    lw              t4, 16(a6)
+    add             t1, t1, s0
+    add             t2, t2, s1
+    add             t0, t0, s2
+    add             t3, t3, s3
+    add             t4, t4, s4
+    sw              t0, (a6)
+    sw              t1, 4(a6)
+    sw              t2, 8(a6)
+    sw              t3, 12(a6)
+    sw              t4, 16(a6)
+
+    ld              s0, (sp)
+    ld              s1, 8(sp)
+    ld              s2, 16(sp)
+    ld              s3, 24(sp)
+    ld              s4, 32(sp)
+    ld              s5, 40(sp)
+    ld              s6, 48(sp)
+    ld              s7, 56(sp)
+    addi            sp, sp, 64
+    ret
+endfunc

x265-4.2.tar/source/common/riscv64/ssd-a.S Added

@@ -0,0 +1,413 @@
+/*****************************************************************************
+ * Copyright (C) 2025 MulticoreWare, Inc
+ *
+ * Authors: Yujiao He <he.yujiao@sanechips.com.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "asm.S"
+
+#ifdef __APPLE__
+.section __RODATA,__rodata
+#else
+.section .rodata
+#endif
+
+.align 4
+
+.text
+
+/* sse_ss, 1 row */
+.macro SSE_SS_x1 s0,s1
+    vle16.v             v8, (\s0)
+    vle16.v             v12, (\s1)
+    vsub.vv             v16, v8, v12
+    vwmacc.vv           v0, v16, v16     
+.endm
+
+/* sse_ss_func, width <= 16, fully unrolled */
+.macro SSE_SS_FUNC_SMALL w,h,lmul1,lmul2
+function PFX(pixel_sse_ss_\w\()x\h\()_rvv)
+    li                  t0, \w
+    slli                a1, a1, 1
+    slli                a3, a3, 1
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    SSE_SS_x1           a0, a2
+.rept \h - 1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    SSE_SS_x1           a0, a2
+.endr
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v8, 0
+#if BIT_DEPTH == 12
+    vwredsum.vs         v8, v0, v8
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+#else
+    vredsum.vs          v8, v0, v8
+#endif
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_SS_FUNC_SMALL 4, 4, m1, m1
+SSE_SS_FUNC_SMALL 8, 8, m2, m1
+SSE_SS_FUNC_SMALL 16, 16, m4, m2
+
+/* ssd_s_func, width <= 16, fully unrolled */
+.macro SSD_S_FUNC_SMALL w,h,lmul1,lmul2
+function PFX(pixel_ssd_s_\w\()x\h\()_rvv)
+    li                  t0, \w
+    slli                a1, a1, 1
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    vle16.v             v8, (a0)
+    vwmacc.vv           v0, v8, v8
+.rept \h - 1
+    add                 a0, a0, a1
+    vle16.v             v8, (a0)
+    vwmacc.vv           v0, v8, v8
+.endr
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSD_S_FUNC_SMALL 4, 4, m1, m1
+SSD_S_FUNC_SMALL 8, 8, m2, m1
+SSD_S_FUNC_SMALL 16, 16, m4, m2
+
+# if !HIGH_BIT_DEPTH
+
+/* sse_pp, 1 row */
+.macro SSE_PP_x1 s0,s1,lmul1,lmul2
+    vsetvli             zero, t0, e8, \lmul2, ta, ma
+    vle8.v              v8, (\s0)
+    vle8.v              v10, (\s1)
+    VABDU               v16, v8, v10, v12
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vzext.vf4           v24, v16
+    vmacc.vv            v0, v24, v24
+.endm
+
+/* sse_pp_func */
+.macro SSE_PP_FUNC w,h,lmul1,lmul2,len
+function PFX(pixel_sse_pp_\w\()x\h\()_rvv)
+    li                  t0, \len
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+.if \w <=16 /* width <=16, fully unrolled */
+    SSE_PP_x1           a0, a2,\lmul1,\lmul2
+.rept \h - 1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    SSE_PP_x1           a0, a2, \lmul1, \lmul2
+.endr
+
+.else /* width >16, loop by row */
+    li                  t1, \h
+1:
+    SSE_PP_x1           a0, a2, \lmul1, \lmul2
+
+.if \w ==64
+    addi                t2, a0, 32
+    addi                t3, a2, 32
+    SSE_PP_x1           t2, t3, \lmul1, \lmul2
+.endif
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    bnez                t1, 1b
+.endif
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_PP_FUNC 4, 4, m1, m1, 4
+SSE_PP_FUNC 4, 8, m1, m1, 4
+SSE_PP_FUNC 8, 8, m2, m1, 8
+SSE_PP_FUNC 8, 16, m2, m1, 8
+SSE_PP_FUNC 16, 16 ,m4, m1, 16
+SSE_PP_FUNC 16, 32, m4, m1, 16
+SSE_PP_FUNC 32, 32, m8, m2, 32
+SSE_PP_FUNC 32, 64, m8, m2, 32
+SSE_PP_FUNC 64, 64, m8, m2, 32
+
+/* sse_ss, width >16, loop by row */
+.macro SSE_SS_FUNC_BIG w,h,lmul1,lmul2
+function PFX(pixel_sse_ss_\w\()x\h\()_rvv)
+    li                  t0, 32
+    slli                a1, a1, 1
+    slli                a3, a3, 1
+    li                  t1, \h
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+    vsetvli             zero, t0, e16, \lmul2, ta, ma       
+1:
+    SSE_SS_x1           a0, a2
+.if \w ==64
+    addi                t2, a0, 64
+    addi                t3, a2, 64
+    SSE_SS_x1           t2, t3
+.endif
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    bnez                t1, 1b
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_SS_FUNC_BIG 32, 32, m8, m4
+SSE_SS_FUNC_BIG 64, 64, m8, m4
+
+/* ssd_s, width >16, loop by row */
+.macro SSD_S_FUNC_BIG w,h,lmul1,lmul2,len
+function PFX(pixel_ssd_s_\w\()x\h\()_rvv)
+    li                  t0, 32
+    slli                a1, a1, 1
+    li                  t1, \h
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+1:
+    vle16.v             v8, (a0)           
+    vwmacc.vv           v0, v8, v8
+.if \w == 64
+    addi                t2, a0, 64
+    vle16.v             v8, (t2)
+    vwmacc.vv           v0, v8, v8
+.endif      
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    bnez                t1, 1b
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSD_S_FUNC_BIG 32, 32, m8, m4
+SSD_S_FUNC_BIG 64, 64, m8, m4
+
+#else // !HIGH BIT DEPTH
+
+/* sse_pp, 1 row, width <=16 */
+.macro SSE_PP_x1_SMALL s0,s1,lmul1,lmul2
+    vle16.v             v8, (\s0)
+    vle16.v             v10, (\s1)
+    VABDU               v16, v8, v10, v12
+    vwmacc.vv           v0, v16, v16
+.endm
+
+/* sse_pp_func, width <=16 */
+.macro SSE_PP_FUNC_SMALL w,h,lmul1,lmul2
+function PFX(pixel_sse_pp_\w\()x\h\()_rvv)
+    li                  t0, \w
+    slli                a1, a1, 1
+    slli                a3, a3, 1
+
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    SSE_PP_x1_SMALL     a0, a2, \lmul1, \lmul2
+.rept \h - 1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    SSE_PP_x1_SMALL     a0, a2, \lmul1, \lmul2
+.endr
+    vsetvli             zero, t0, e32, \lmul1, ta, ma
+    vmv.v.i             v8, 0
+#if BIT_DEPTH == 12
+    vwredsum.vs         v8, v0, v8
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+#else
+    vredsum.vs          v8, v0, v8
+#endif
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_PP_FUNC_SMALL 4, 4, m1, m1
+SSE_PP_FUNC_SMALL 4, 8, m1, m1
+SSE_PP_FUNC_SMALL 8, 8, m2, m1
+SSE_PP_FUNC_SMALL 8, 16, m2, m1
+SSE_PP_FUNC_SMALL 16, 16, m4, m2
+SSE_PP_FUNC_SMALL 16, 32, m4, m2
+
+/* sse_pp, 1 row, width > 16 */
+.macro SSE_PP_x1_BIG s0,s1,lmul1,lmul2
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    vle16.v             v8, (\s0)
+    vle16.v             v10, (\s1)
+    VABDU               v16, v8, v10, v12
+
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vzext.vf4           v24, v16
+    vmacc.vv            v0, v24, v24
+.endm
+
+/* sse_pp_func,  width >16 */
+.macro SSE_PP_FUNC_BIG w,h,lmul1,lmul2
+function PFX(pixel_sse_pp_\w\()x\h\()_rvv)
+    li                  t0, 16
+    slli                a1, a1, 1
+    slli                a3, a3, 1
+    li                  t1, \h
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+1:
+    mv                  t2, a0
+    mv                  t3, a2
+    SSE_PP_x1_BIG       a0, a2, \lmul1, \lmul2
+.rept (\w / 16) -1
+    addi                t2, t2, 32
+    addi                t3, t3, 32
+    SSE_PP_x1_BIG       t2, t3, \lmul1, \lmul2
+.endr
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    bnez                t1, 1b
+
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_PP_FUNC_BIG 32, 32, m8, m2
+SSE_PP_FUNC_BIG 32, 64, m8, m2
+SSE_PP_FUNC_BIG 64, 64, m8, m2
+
+/* sse_ss, 1 row, width >16 */
+.macro SSE_SS_x1_BIG s0,s1,lmul1,lmul2
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    vle16.v             v8, (\s0)
+    vle16.v             v10, (\s1)
+    vsub.vv             v16, v8, v10
+
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vsext.vf4           v8, v16
+    vmacc.vv            v0, v8, v8
+.endm
+
+/* sse_ss_func, width > 16, loop by row */
+.macro SSE_SS_FUNC_BIG w,h,lmul1,lmul2
+function PFX(pixel_sse_ss_\w\()x\h\()_rvv)
+    li                  t0, 16
+    slli                a1, a1, 1
+    slli                a3, a3, 1
+
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+    li                  t1, \h
+1:
+    mv                  t2, a0
+    mv                  t3, a2
+    SSE_SS_x1_BIG       a0, a2, \lmul1, \lmul2
+.rept (\w / 16) -1
+    addi                t2, t2, 32
+    addi                t3, t3, 32
+    SSE_SS_x1_BIG       t2, t3, \lmul1, \lmul2
+.endr
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    add                 a2, a2, a3
+    bnez                t1, 1b
+
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSE_SS_FUNC_BIG 32, 32, m8, m2
+SSE_SS_FUNC_BIG 64, 64, m8, m2
+
+/* ssd_s, 1 row, width >16 */
+.macro SSD_S_x1_BIG s0,lmul1,lmul2
+    vsetvli             zero, t0, e16, \lmul2, ta, ma
+    vle16.v             v8, (\s0)            
+
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vsext.vf4           v16, v8
+    vmacc.vv            v0, v16, v16        
+.endm
+
+/* ssd_s_func, width > 16, loop by row */
+.macro SSD_S_FUNC_BIG w,h,lmul1,lmul2
+function PFX(pixel_ssd_s_\w\()x\h\()_rvv)
+    li                  t0, 16
+    slli                a1, a1, 1
+    li                  t1, \h
+    vsetvli             zero, t0, e64, \lmul1, ta, ma
+    vmv.v.i             v0, 0
+1:
+    mv                  t2, a0
+    SSD_S_x1_BIG        a0, \lmul1, \lmul2
+.rept (\w / 16) -1
+    addi                t2, t2, 32
+    SSD_S_x1_BIG        t2, \lmul1, \lmul2
+.endr     
+    addi                t1, t1, -1
+    add                 a0, a0, a1
+    bnez                t1, 1b
+
+    vmv.v.i             v8, 0
+    vredsum.vs          v8, v0, v8
+    vmv.x.s             a0, v8
+    ret
+endfunc
+.endm
+
+SSD_S_FUNC_BIG 32, 32, m8, m2
+SSD_S_FUNC_BIG 64, 64, m8, m2
+
+#endif

x265-4.1.tar/source/common/scaler.cpp -> x265-4.2.tar/source/common/scaler.cpp Changed

x265-4.1.tar/source/common/slice.h -> x265-4.2.tar/source/common/slice.h Changed

@@ -26,6 +26,7 @@
 #define X265_SLICE_H
 
 #include "common.h"
+#include "mv.h"
 
 namespace X265_NS {
 // private namespace
@@ -35,6 +36,8 @@
 class PicYuv;
 class MotionReference;
 
+struct MEData;
+
 enum SliceType
 {
     B_SLICE,
@@ -104,6 +107,10 @@
         LEVEL6 = 180,
         LEVEL6_1 = 183,
         LEVEL6_2 = 186,
+        LEVEL6_3 = 189,
+        LEVEL7 = 210,
+        LEVEL7_1 = 213,
+        LEVEL7_2 = 216,
         LEVEL8_5 = 255,
     };
 }
@@ -113,7 +120,7 @@
     int      profileIdcMAX_LAYERS;
     int      levelIdc;
     uint32_t minCrForLevel;
-    uint32_t maxLumaSrForLevel;
+    uint64_t maxLumaSrForLevel;
     uint32_t bitDepthConstraint;
     int      chromaFormatConstraint;
     bool     tierFlag;
@@ -374,6 +381,7 @@
     WeightParam m_weightPredTable2MAX_NUM_REF3; // listrefIdx0:Y, 1:U, 2:V
     MotionReference (*m_mref)MAX_NUM_REF + 1;
     RPS         m_rps;
+    MEData*     m_ctuMV;
 
     NalUnitType m_nalUnitType;
     SliceType   m_sliceType;
@@ -415,6 +423,7 @@
         m_lastIDR = 0;
         m_sLFaseFlag = true;
         m_numRefIdx0 = m_numRefIdx1 = 0;
+        m_ctuMV = NULL;
         memset(m_refFrameList, 0, sizeof(m_refFrameList));
         memset(m_refReconPicList, 0, sizeof(m_refReconPicList));
         memset(m_refPOCList, 0, sizeof(m_refPOCList));

x265-4.1.tar/source/common/threading.cpp -> x265-4.2.tar/source/common/threading.cpp Changed

x265-4.1.tar/source/common/threading.h -> x265-4.2.tar/source/common/threading.h Changed

@@ -56,15 +56,20 @@
 int no_atomic_inc(int* ptr);
 int no_atomic_dec(int* ptr);
 int no_atomic_add(int* ptr, int val);
+int64_t no_atomic_add64(int64_t* ptr, int64_t val);
 }
 
-#define CLZ(id, x)            id = (unsigned long)__builtin_clz(x) ^ 31
-#define CTZ(id, x)            id = (unsigned long)__builtin_ctz(x)
+#define BSR(id, x)            (id) = ((unsigned long)__builtin_clz(x) ^ 31)
+#define BSF(id, x)            (id) = ((unsigned long)__builtin_ctz(x))
+#define BSR64(id, x)          (id) = ((unsigned long)__builtin_clzll(x) ^ 63)
+#define BSF64(id, x)          (id) = ((unsigned long)__builtin_ctzll(x))
 #define ATOMIC_OR(ptr, mask)  no_atomic_or((int*)ptr, mask)
 #define ATOMIC_AND(ptr, mask) no_atomic_and((int*)ptr, mask)
 #define ATOMIC_INC(ptr)       no_atomic_inc((int*)ptr)
 #define ATOMIC_DEC(ptr)       no_atomic_dec((int*)ptr)
-#define ATOMIC_ADD(ptr, val)  no_atomic_add((int*)ptr, val)
+#define ATOMIC_ADD(ptr, val)  (sizeof(*(ptr)) == 8 ? \
+                               no_atomic_add64((int64_t*)ptr, (int64_t)(val)) : \
+                               no_atomic_add((int*)ptr, (int)(val)))
 #define GIVE_UP_TIME()        usleep(0)
 
 #elif __GNUC__               /* GCCs builtin atomics */
@@ -72,24 +77,30 @@
 #include <sys/time.h>
 #include <unistd.h>
 
-#define CLZ(id, x)            id = (unsigned long)__builtin_clz(x) ^ 31
-#define CTZ(id, x)            id = (unsigned long)__builtin_ctz(x)
+#define BSR(id, x)            (id) = ((unsigned long)__builtin_clz(x) ^ 31)
+#define BSF(id, x)            (id) = ((unsigned long)__builtin_ctz(x))
+#define BSR64(id, x)          (id) = ((unsigned long)__builtin_clzll(x) ^ 63)
+#define BSF64(id, x)          (id) = ((unsigned long)__builtin_ctzll(x))
 #define ATOMIC_OR(ptr, mask)  __sync_fetch_and_or(ptr, mask)
 #define ATOMIC_AND(ptr, mask) __sync_fetch_and_and(ptr, mask)
 #define ATOMIC_INC(ptr)       __sync_add_and_fetch((volatile int32_t*)ptr, 1)
 #define ATOMIC_DEC(ptr)       __sync_add_and_fetch((volatile int32_t*)ptr, -1)
-#define ATOMIC_ADD(ptr, val)  __sync_fetch_and_add((volatile int32_t*)ptr, val)
+#define ATOMIC_ADD(ptr, val)  __sync_fetch_and_add((volatile __typeof__(*(ptr))*)ptr, (__typeof__(*(ptr) + 0))(val))
 #define GIVE_UP_TIME()        usleep(0)
 
 #elif defined(_MSC_VER)       /* Windows atomic intrinsics */
 
 #include <intrin.h>
 
-#define CLZ(id, x)            _BitScanReverse(&id, x)
-#define CTZ(id, x)            _BitScanForward(&id, x)
+#define BSR(id, x)            _BitScanReverse(&id, x)
+#define BSF(id, x)            _BitScanForward(&id, x)
+#define BSR64(id, x)          _BitScanReverse64(&id, x)
+#define BSF64(id, x)          _BitScanForward64(&id, x)
 #define ATOMIC_INC(ptr)       InterlockedIncrement((volatile LONG*)ptr)
 #define ATOMIC_DEC(ptr)       InterlockedDecrement((volatile LONG*)ptr)
-#define ATOMIC_ADD(ptr, val)  InterlockedExchangeAdd((volatile LONG*)ptr, val)
+#define ATOMIC_ADD(ptr, val)  (sizeof(*(ptr)) == 8 ? \
+                               InterlockedExchangeAdd64((volatile LONGLONG*)ptr, (LONGLONG)(val)) : \
+                               InterlockedExchangeAdd((volatile LONG*)ptr, (LONG)(val)))
 #define ATOMIC_OR(ptr, mask)  _InterlockedOr((volatile LONG*)ptr, (LONG)mask)
 #define ATOMIC_AND(ptr, mask) _InterlockedAnd((volatile LONG*)ptr, (LONG)mask)
 #define GIVE_UP_TIME()        Sleep(0)

x265-4.1.tar/source/common/threadpool.cpp -> x265-4.2.tar/source/common/threadpool.cpp Changed

@@ -27,22 +27,32 @@
 #include "threading.h"
 
 #include <new>
+#include <vector>
 
 #if defined(_WIN32_WINNT) && _WIN32_WINNT >= _WIN32_WINNT_WIN7
 #include <winnt.h>
 #endif
 
+#if defined(__APPLE__)
+#include <sys/sysctl.h>
+#elif !defined(_WIN32)
+#include <fstream>
+#include <string>
+#include <cstdio>
+#include <cstdlib>
+#endif
+
 #if X86_64
 
 #ifdef __GNUC__
 
-#define SLEEPBITMAP_CTZ(id, x)     id = (unsigned long)__builtin_ctzll(x)
+#define SLEEPBITMAP_BSF(id, x)     (id) = ((unsigned long)__builtin_ctzll(x))
 #define SLEEPBITMAP_OR(ptr, mask)  __sync_fetch_and_or(ptr, mask)
 #define SLEEPBITMAP_AND(ptr, mask) __sync_fetch_and_and(ptr, mask)
 
 #elif defined(_MSC_VER)
 
-#define SLEEPBITMAP_CTZ(id, x)     _BitScanForward64(&id, x)
+#define SLEEPBITMAP_BSF(id, x)     _BitScanForward64(&id, x)
 #define SLEEPBITMAP_OR(ptr, mask)  InterlockedOr64((volatile LONG64*)ptr, (LONG)mask)
 #define SLEEPBITMAP_AND(ptr, mask) InterlockedAnd64((volatile LONG64*)ptr, (LONG)mask)
 
@@ -51,7 +61,7 @@
 #else
 
 /* use 32-bit primitives defined in threading.h */
-#define SLEEPBITMAP_CTZ CTZ
+#define SLEEPBITMAP_BSF BSF
 #define SLEEPBITMAP_OR  ATOMIC_OR
 #define SLEEPBITMAP_AND ATOMIC_AND
 
@@ -206,7 +216,7 @@
     sleepbitmap_t masked = m_sleepBitmap & firstTryBitmap;
     while (masked)
     {
-        SLEEPBITMAP_CTZ(id, masked);
+        SLEEPBITMAP_BSF(id, masked);
 
         sleepbitmap_t bit = (sleepbitmap_t)1 << id;
         if (SLEEPBITMAP_AND(&m_sleepBitmap, ~bit) & bit)
@@ -218,7 +228,7 @@
     masked = m_sleepBitmap & secondTryBitmap;
     while (masked)
     {
-        SLEEPBITMAP_CTZ(id, masked);
+        SLEEPBITMAP_BSF(id, masked);
 
         sleepbitmap_t bit = (sleepbitmap_t)1 << id;
         if (SLEEPBITMAP_AND(&m_sleepBitmap, ~bit) & bit)
@@ -247,6 +257,135 @@
 
     return bondCount;
 }
+
+/* Distributes totalNumThreads between ThreadedME and FrameEncoder pools.
+ * Modifies threadsPerPool, nodeMaskPerPool, numNumaNodes, and numPools in-place.
+ * Returns the number of threads reserved for frame encoding. */
+static void distributeThreadsForTme(
+    x265_param* p,
+    int totalNumThreads,
+    int& numNumaNodes,
+    bool bNumaSupport,
+    int* threadsPerPool,
+    uint64_t* nodeMaskPerPool,
+    int& numPools,
+    int& threadsFrameEnc)
+{
+    if (totalNumThreads < MIN_TME_THREADS)
+    {
+        x265_log(p, X265_LOG_WARNING, "Low thread count detected, disabling --threaded-me."
+            " Minimum recommended is 32 cores / threads\n");
+        p->bThreadedME = 0;
+        return;
+    }
+
+    int targetTME = ThreadPool::configureTmeThreadCount(p, totalNumThreads);
+    targetTME = (targetTME < 1) ? 1 : targetTME;
+
+    threadsFrameEnc = totalNumThreads - targetTME;
+    int defaultNumFT = ThreadPool::getFrameThreadsCount(p, totalNumThreads);
+    if (threadsFrameEnc < defaultNumFT)
+    {
+        threadsFrameEnc = defaultNumFT;
+        targetTME = totalNumThreads - threadsFrameEnc;
+    }
+
+#if defined(_WIN32_WINNT) && _WIN32_WINNT >= _WIN32_WINNT_WIN7 || HAVE_LIBNUMA
+    if (bNumaSupport && numNumaNodes > 1)
+    {
+        int tmeNumaNodes = 0;
+        int leftover = 0;
+
+        // First thread pool belongs to ThreadedME
+        std::vector<int> threads(1, 0);
+        std::vector<uint64_t> nodeMasks(1, 0);
+        int poolIndex = 0;
+
+        /* Greedily assign whole NUMA nodes to TME until reaching or exceeding the target */
+        for (int i = 0; i < numNumaNodes + 1; i++)
+        {
+            if (!threadsPerPooli && !nodeMaskPerPooli)
+                continue;
+
+            int toTake = X265_MIN(threadsPerPooli, targetTME - threads0);
+            if (toTake > 0)
+            {
+                threadspoolIndex += toTake;
+                nodeMaskspoolIndex |= nodeMaskPerPooli;
+                tmeNumaNodes++;
+
+                if (threads0 == targetTME)
+                    poolIndex++;
+
+                if (toTake < threadsPerPooli)
+                    leftover = threadsPerPooli - toTake;
+            }
+            else
+            {
+                threads.push_back(threadsPerPooli);
+                nodeMasks.push_back(nodeMaskPerPooli);
+                poolIndex++;
+            }
+        }
+
+        // Distribute leftover threads among FrameEncoders
+        if (leftover)
+        {
+            // Case 1: There are 1 or more threadpools for FrameEncoder(s) by now
+            if (threads.size() > 1)
+            {
+                int split = static_cast<int>(static_cast<double>(leftover) / (numNumaNodes - 1));
+                for (int pool = 1; pool < numNumaNodes; pool++)
+                {
+                    int give = X265_MIN(split, leftover);
+                    threadspool += give;
+                    leftover -= give;
+                }
+            }
+
+            // Case 2: FrameEncoder(s) haven't received threads yet
+            if (threads.size() == 1)
+            {
+                threads.push_back(leftover);
+                // Give the same node mask as the last node of ThreadedME
+                uint64_t msb = 1;
+                uint64_t tmeNodeMask = nodeMasks0;
+                while (tmeNodeMask > 1)
+                {
+                    tmeNodeMask >>= 1;
+                    msb <<= 1;
+                }
+                nodeMasks.push_back(msb);
+            }
+        }
+
+        // Apply calculated threadpool assignment
+        memset(threadsPerPool, 0, sizeof(int) * (numNumaNodes + 2));
+        memset(nodeMaskPerPool, 0, sizeof(uint64_t) * (numNumaNodes + 2));
+
+        numPools = numNumaNodes = static_cast<int>(threads.size());
+        for (int pool = 0; pool < numPools; pool++)
+        {
+            threadsPerPoolpool = threadspool;
+            nodeMaskPerPoolpool = nodeMaskspool;
+        }
+    }
+    else
+#endif
+    {
+        memset(threadsPerPool, 0, sizeof(int) * (numNumaNodes + 2));
+        memset(nodeMaskPerPool, 0, sizeof(uint64_t) * (numNumaNodes + 2));
+
+        threadsPerPool0 = targetTME;
+        nodeMaskPerPool0 = 1;
+
+        threadsPerPool1 = threadsFrameEnc;
+        nodeMaskPerPool1 = 1;
+
+        numPools = 2;
+    }
+}
+
 ThreadPool* ThreadPool::allocThreadPools(x265_param* p, int& numPools, bool isThreadsReserved)
 {
     enum { MAX_NODE_NUM = 127 };
@@ -373,6 +512,22 @@
             nodeMaskPerPoolnumNumaNodes |= ((uint64_t)1 << i);
         }
     }
+
+    /* If ThreadedME is enabled, split resources: give half NUMA nodes (or
+     * half cores when NUMA not available) to threaded-me and the other half
+     * to the remaining job providers. */
+    /* Compute total CPU count from detected per-node counts when available */
+    for (int i = 0; i < numNumaNodes + 1; i++)
+        totalNumThreads += threadsPerPooli;
+    if (!totalNumThreads)
+        totalNumThreads = ThreadPool::getCpuCount();
+
+    int threadsFrameEnc = totalNumThreads;
+    if (p->bThreadedME)
+    {
+        distributeThreadsForTme(p, totalNumThreads, numNumaNodes, bNumaSupport, threadsPerPool,
+                                nodeMaskPerPool, numPools, threadsFrameEnc);
+    }
  
     // If the last pool size is > MAX_POOL_THREADS, clip it to spawn thread pools only of size >= 1/2 max (heuristic)
     if ((threadsPerPoolnumNumaNodes > MAX_POOL_THREADS) &&
@@ -383,17 +538,22 @@
                  "Creating only %d worker threads beyond specified numbers with --pools (if specified) to prevent asymmetry in pools; may not use all HW contexts\n", threadsPerPoolnumNumaNodes);
     }
 
-    numPools = 0;
-    for (int i = 0; i < numNumaNodes + 1; i++)
+    if (!p->bThreadedME)
     {
-        if (bNumaSupport)
-            x265_log(p, X265_LOG_DEBUG, "NUMA node %d may use %d logical cores\n", i, cpusPerNodei);
-        if (threadsPerPooli)
+        numPools = 0;
+        for (int i = 0; i < numNumaNodes + 1; i++)
         {
-            numPools += (threadsPerPooli + MAX_POOL_THREADS - 1) / MAX_POOL_THREADS;
-            totalNumThreads += threadsPerPooli;
+            if (bNumaSupport)
+                x265_log(p, X265_LOG_DEBUG, "NUMA node %d may use %d logical cores\n", i, cpusPerNodei);
+            
+            if (threadsPerPooli)
+            {
+                numPools += (threadsPerPooli + MAX_POOL_THREADS - 1) / MAX_POOL_THREADS;
+                totalNumThreads += threadsPerPooli;
+            }
         }
     }
+        
     if (!isThreadsReserved)
     {
         if (!numPools)
@@ -403,13 +563,13 @@
         }
 
         if (!p->frameNumThreads)
-            ThreadPool::getFrameThreadsCount(p, totalNumThreads);
+            p->frameNumThreads = ThreadPool::getFrameThreadsCount(p, threadsFrameEnc);
     }
     
     if (!numPools)
         return NULL;
 
-    if (numPools > p->frameNumThreads)
+    if (numPools > p->frameNumThreads && !p->bThreadedME)
     {
         x265_log(p, X265_LOG_DEBUG, "Reducing number of thread pools for frame thread count\n");
         numPools = X265_MAX(p->frameNumThreads / 2, 1);
@@ -419,30 +579,36 @@
     ThreadPool *pools = new ThreadPoolnumPools;
     if (pools)
     {
-        int maxProviders = (p->frameNumThreads + numPools - 1) / numPools + !isThreadsReserved; /* +1 is Lookahead, always assigned to threadpool 0 */
+        int poolCount = (p->bThreadedME) ? numPools - 1 : numPools;
         int node = 0;
         for (int i = 0; i < numPools; i++)
         {
+            int maxProviders = (p->bThreadedME && i == 0) // threadpool 0 is dedicated to ThreadedME
+                ? 1
+                : (p->frameNumThreads + poolCount - 1) / poolCount + !isThreadsReserved; // +1 is Lookahead, always assigned to threadpool 0
+            
             while (!threadsPerPoolnode)
                 node++;
-            int numThreads = X265_MIN(MAX_POOL_THREADS, threadsPerPoolnode);
+            int numThreads = threadsPerPoolnode;
             int origNumThreads = numThreads;
+
             if (i == 0 && p->lookaheadThreads > numThreads / 2)
             {
                 p->lookaheadThreads = numThreads / 2;
                 x265_log(p, X265_LOG_DEBUG, "Setting lookahead threads to a maximum of half the total number of threads\n");
             }
+
             if (isThreadsReserved)
             {
                 numThreads = p->lookaheadThreads;
                 maxProviders = 1;
             }
-
             else if (i == 0)
                 numThreads -= p->lookaheadThreads;
+            
             if (!poolsi.create(numThreads, maxProviders, nodeMaskPerPoolnode))
             {
-                X265_FREE(pools);
+                delete pools;
                 numPools = 0;
                 return NULL;
             }
@@ -510,6 +676,8 @@
             new (m_workers + i)WorkerThread(*this, i);
 
     m_jpTable = X265_MALLOC(JobProvider*, maxProviders);
+    if (m_jpTable)
+        memset(m_jpTable, 0, sizeof(JobProvider*) * maxProviders);
     m_numProviders = 0;
 
     return m_workers && m_jpTable;
@@ -659,25 +827,173 @@
 #endif
 }
 
-void ThreadPool::getFrameThreadsCount(x265_param* p, int cpuCount)
+int ThreadPool::getFrameThreadsCount(x265_param* p, int cpuCount)
 {
     int rows = (p->sourceHeight + p->maxCUSize - 1) >> g_log2Sizep->maxCUSize;
     if (!p->bEnableWavefront)
-        p->frameNumThreads = X265_MIN3(cpuCount, (rows + 1) / 2, X265_MAX_FRAME_THREADS);
+        return X265_MIN3(cpuCount, (rows + 1) / 2, X265_MAX_FRAME_THREADS);
     else if (cpuCount >= 32)
-        p->frameNumThreads = (p->sourceHeight > 2000) ? 6 : 5; 
+        return (p->sourceHeight > 2000) ? 6 : 5; 
     else if (cpuCount >= 16)
-        p->frameNumThreads = 4; 
+        return 4; 
     else if (cpuCount >= 8)
 #if _WIN32 && X265_ARCH_ARM64
-        p->frameNumThreads = cpuCount;
+        return cpuCount;
 #else
-        p->frameNumThreads = 3;
+        return 3;
 #endif
     else if (cpuCount >= 4)
-        p->frameNumThreads = 2;
+        return 2;
     else
-        p->frameNumThreads = 1;
+        return 1;
+}
+
+int ThreadPool::configureTmeThreadCount(x265_param* param, int cpuCount)
+{
+    enum TmeResClass
+    {
+        TME_RES_LOW = 0,
+        TME_RES_MID,
+        TME_RES_HIGH,
+        TME_RES_COUNT
+    };
+
+    enum TmeRule
+    {
+        TME_RULE_SLOWER_VERYSLOW = 0,
+        TME_RULE_SLOW,
+        TME_RULE_FAST_MEDIUM,
+        TME_RULE_FASTER,
+        TME_RULE_VERYFAST,
+        TME_RULE_SUPERFAST,
+        TME_RULE_ULTRAFAST,
+        TME_RULE_COUNT
+    };
+
+    struct TmeRuleConfig
+    {
+        int taskBlockSizeTME_RES_COUNT;
+        int numBufferRowsTME_RES_COUNT;
+        int threadPercentTME_RES_COUNT;
+        bool widthBasedTaskBlockSize;
+    };
+
+    static const TmeRuleConfig s_tmeRuleConfigTME_RULE_COUNT =
+    {
+        { { 1, 1, 1 }, { 3, 3, 3 }, { 90, 85, 70 }, false }, // slower / veryslow preset and similar options
+        { { 1, 1, 1 }, { 5, 5, 3 }, { 90, 85, 75 }, false }, // slow preset and similar options
+        { { 1, 1, 1 }, { 10, 10, 10 }, { 90, 80, 70 }, false }, // fast / medium preset and similar options
+        { { 1, 1, 1 }, { 10, 15, 10 }, { 90, 80, 70 }, false }, // faster preset and similar options
+        { { 1, 1, 1 }, { 10, 15, 20 }, { 90, 80, 70 }, false }, // veryfast preset and similar options
+        { { 2, 4, 4 }, { 10, 15, 20 }, { 90, 80, 60 }, false }, // superfast preset and similar options
+        { { 0, 0, 0 }, { 15, 20, 20 }, { 90, 80, 50 }, true  }  // ultrafast preset and similar options
+    };
+
+    const int resClass = (param->sourceHeight >= 1440) ? TME_RES_HIGH :
+                         (param->sourceHeight <= 720) ? TME_RES_LOW : TME_RES_MID;
+
+    const bool ruleMatchesTME_RULE_COUNT =
+    {
+        param->maxNumReferences >= 5 || param->subpelRefine >= 4 || (param->bEnableRectInter && param->bEnableAMP),
+        param->maxNumReferences >= 4 || param->subpelRefine >= 3 || (param->bEnableRectInter ^ param->bEnableAMP),
+        param->maxNumReferences >= 3 && param->subpelRefine >= 2,
+        param->maxNumReferences >= 2 && param->subpelRefine >= 2,
+        param->subpelRefine >= 1 && param->bframes > 3,
+        param->subpelRefine && param->maxCUSize < 64,
+        !param->subpelRefine || param->searchMethod == X265_DIA_SEARCH || param->minCUSize >= 16
+    };
+
+    int selectedRule = -1;
+    for (int i = 0; i < TME_RULE_COUNT; i++)
+    {
+        if (ruleMatchesi)
+        {
+            selectedRule = i;
+            break;
+        }
+    }
+
+    bool isHighFreq = (getCPUFrequencyMHz() > 2000.0);
+
+    if (selectedRule >= 0)
+    {
+        const TmeRuleConfig& cfg = s_tmeRuleConfigselectedRule;
+        param->tmeTaskBlockSize = cfg.widthBasedTaskBlockSize ? ((param->sourceWidth + 480 - 1) / 480) : cfg.taskBlockSizeresClass;
+        param->tmeNumBufferRows = cfg.numBufferRowsresClass;
+        return (!isHighFreq) ? (cpuCount * cfg.threadPercentresClass) / 100 : cpuCount / 2;
+    }
+
+    static const int s_defaultThreadPercentTME_RES_COUNT = { 80, 80, 70 };
+    return (!isHighFreq) ? (cpuCount * s_defaultThreadPercentresClass) / 100 : cpuCount / 2;
+}
+
+double getCPUFrequencyMHz()
+{
+#if defined(_WIN32)
+    HKEY hKey;
+    DWORD mhz = 0;
+    DWORD size = sizeof(mhz);
+    if (RegOpenKeyExA(HKEY_LOCAL_MACHINE,
+                      "HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0",
+                      0, KEY_READ, &hKey) == ERROR_SUCCESS)
+    {
+        RegQueryValueExA(hKey, "~MHz", NULL, NULL, (LPBYTE)&mhz, &size);
+        RegCloseKey(hKey);
+    }
+    return (double)mhz;
+
+#elif defined(__APPLE__)
+    uint64_t freq = 0;
+    size_t size = sizeof(freq);
+    if (sysctlbyname("hw.cpufrequency", &freq, &size, NULL, 0) == 0)
+        return (double)freq / 1.0e6;
+    return 0.0;
+
+#else  /* Linux */
+    /* scaling_cur_freq reflects the live frequency chosen by the governor
+     * and EPP hint. Iterate over all cpuN entries and return the highest observed value.
+     */
+    {
+        uint64_t maxKhz = 0;
+        char path64;
+        for (int cpu = 0; ; ++cpu)
+        {
+            snprintf(path, sizeof(path),
+                     "/sys/devices/system/cpu/cpu%d/cpufreq/scaling_cur_freq", cpu);
+            std::ifstream f(path);
+            if (!f.is_open())
+                break;
+            uint64_t khz = 0;
+            f >> khz;
+            if (khz > maxKhz)
+                maxKhz = khz;
+        }
+        if (maxKhz > 0)
+            return (double)maxKhz / 1000.0;
+    }
+    /* Fall back to /proc/cpuinfo — collect the max "cpu MHz" across all entries. */
+    {
+        std::ifstream f("/proc/cpuinfo");
+        std::string line;
+        double maxMhz = 0.0;
+        while (std::getline(f, line))
+        {
+            if (line.find("cpu MHz") != std::string::npos)
+            {
+                size_t colon = line.find(':');
+                if (colon != std::string::npos)
+                {
+                    double mhz = strtod(line.c_str() + colon + 1, NULL);
+                    if (mhz > maxMhz)
+                        maxMhz = mhz;
+                }
+            }
+        }
+        if (maxMhz > 0.0)
+            return maxMhz;
+    }
+    return 0.0;
+#endif
 }
 
 } // end namespace X265_NS

x265-4.1.tar/source/common/threadpool.h -> x265-4.2.tar/source/common/threadpool.h Changed

@@ -102,10 +102,12 @@
     void setThreadNodeAffinity(void *numaMask);
     int  tryAcquireSleepingThread(sleepbitmap_t firstTryBitmap, sleepbitmap_t secondTryBitmap);
     int  tryBondPeers(int maxPeers, sleepbitmap_t peerBitmap, BondedTaskGroup& master);
+
     static ThreadPool* allocThreadPools(x265_param* p, int& numPools, bool isThreadsReserved);
     static int  getCpuCount();
     static int  getNumaNodeCount();
-    static void getFrameThreadsCount(x265_param* p,int cpuCount);
+    static int  getFrameThreadsCount(x265_param* p, int cpuCount);
+    static int  configureTmeThreadCount(x265_param* p, int cpuCount);
 };
 
 /* Any worker thread may enlist the help of idle worker threads from the same
@@ -169,6 +171,20 @@
     virtual void processTasks(int workerThreadId) = 0;
 };
 
+/**
+ * @brief Return the highest current CPU frequency in MHz across all cores, or 0.0 if unavailable.
+ *
+ * The value reflects the live frequency as reported by the cpufreq subsystem,
+ * which accounts for the active scaling governor and EPP hint.
+ *
+ * Platform support:
+ *   Linux   – iterates /sys/devices/system/cpu/cpuN/cpufreq/scaling_cur_freq (kHz)
+ *              for all cores and returns the maximum; falls back to /proc/cpuinfo
+ *   macOS   – sysctl hw.cpufrequency (Hz)
+ *   Windows – registry ~MHz under CentralProcessor\0
+ */
+double getCPUFrequencyMHz();
+
 } // end namespace X265_NS
 
 #endif // ifndef X265_THREADPOOL_H

x265-4.1.tar/source/common/wavefront.cpp -> x265-4.2.tar/source/common/wavefront.cpp Changed

x265-4.1.tar/source/common/x86/blockcopy8.asm -> x265-4.2.tar/source/common/x86/blockcopy8.asm Changed

x265-4.1.tar/source/common/x86/const-a.asm -> x265-4.2.tar/source/common/x86/const-a.asm Changed

x265-4.1.tar/source/common/x86/cpu-a.asm -> x265-4.2.tar/source/common/x86/cpu-a.asm Changed

x265-4.1.tar/source/common/x86/dct8.asm -> x265-4.2.tar/source/common/x86/dct8.asm Changed

x265-4.1.tar/source/common/x86/h-ipfilter16.asm -> x265-4.2.tar/source/common/x86/h-ipfilter16.asm Changed

x265-4.1.tar/source/common/x86/h-ipfilter8.asm -> x265-4.2.tar/source/common/x86/h-ipfilter8.asm Changed

x265-4.1.tar/source/common/x86/h4-ipfilter16.asm -> x265-4.2.tar/source/common/x86/h4-ipfilter16.asm Changed

x265-4.1.tar/source/common/x86/intrapred16.asm -> x265-4.2.tar/source/common/x86/intrapred16.asm Changed

x265-4.1.tar/source/common/x86/intrapred8.asm -> x265-4.2.tar/source/common/x86/intrapred8.asm Changed

x265-4.1.tar/source/common/x86/intrapred8_allangs.asm -> x265-4.2.tar/source/common/x86/intrapred8_allangs.asm Changed

x265-4.1.tar/source/common/x86/ipfilter16.asm -> x265-4.2.tar/source/common/x86/ipfilter16.asm Changed

x265-4.1.tar/source/common/x86/ipfilter8.asm -> x265-4.2.tar/source/common/x86/ipfilter8.asm Changed

x265-4.1.tar/source/common/x86/loopfilter.asm -> x265-4.2.tar/source/common/x86/loopfilter.asm Changed

x265-4.1.tar/source/common/x86/mc-a.asm -> x265-4.2.tar/source/common/x86/mc-a.asm Changed

x265-4.1.tar/source/common/x86/mc-a2.asm -> x265-4.2.tar/source/common/x86/mc-a2.asm Changed

x265-4.1.tar/source/common/x86/pixel-32.asm -> x265-4.2.tar/source/common/x86/pixel-32.asm Changed

x265-4.1.tar/source/common/x86/pixel-a.asm -> x265-4.2.tar/source/common/x86/pixel-a.asm Changed

x265-4.1.tar/source/common/x86/pixel-util8.asm -> x265-4.2.tar/source/common/x86/pixel-util8.asm Changed

@@ -8484,7 +8484,13 @@
     ; r4 - nonZero
     ; r5 - scanFlagMask
     ; r6 - sum
+
+%if FORMAT_ELF
+    mov         r0, private_prefix %+ _entropyStateBits wrt ..gotpc
+%else
     lea         r0, private_prefix %+ _entropyStateBits
+    ;mov         r0, private_prefix %+ _entropyStateBits
+%endif
     mov         r1, r6mp
     xor         r6d, r6d
     xor         r4d, r4d
@@ -8664,7 +8670,12 @@
     ; r6 - sum
     ; {r3,r4} - ctxSig15-0
     ; r8m - (numNonZero != 0) || (subPosBase == 0)
+%if FORMAT_ELF
+    mov             r0, private_prefix %+ _entropyStateBits wrt ..gotpc
+%else
     lea             r0, private_prefix %+ _entropyStateBits
+    ;mov             r0, private_prefix %+ _entropyStateBits
+%endif
     mov             r1, r6mp
     xor             r6d, r6d
     xor             r8d, r8d
@@ -8753,7 +8764,7 @@
 ;        {
 ;            {
 ;                unsigned long cidx;
-;                CLZ(cidx, codeNumber + 1);
+;                BSR(cidx, codeNumber + 1);
 ;                length = cidx;
 ;            }
 ;            codeNumber = (length + length);
@@ -8812,7 +8823,7 @@
 
     shr         eax, t3b                ; codeNumber = ((uint32_t)codeNumber >> goRiceParam) - COEF_REMAIN_BIN_REDUCTION
 
-    lea         r2d, rax - 3 + 1      ; CLZ(cidx, codeNumber + 1);
+    lea         r2d, rax - 3 + 1      ; BSR(cidx, codeNumber + 1);
     bsr         r2d, r2d
     add         r2d, r2d                ; codeNumber = (length + length)
 
@@ -8903,7 +8914,12 @@
     or          r11d, 0x100                     ; default value setting to 8
     bsf         r11d, r11d
 
-    lea         r5, private_prefix %+ _entropyStateBits
+%if FORMAT_ELF
+    mov             r5, private_prefix %+ _entropyStateBits wrt ..gotpc
+%else
+    lea             r5, private_prefix %+ _entropyStateBits
+    ;mov             r5, private_prefix %+ _entropyStateBits
+%endif
     xor         r6d, r6d
     mov         r4d, 0xFFFFFFF9
 
@@ -8975,3 +8991,4 @@
     or          eax, r4d
     RET
 %endif ; ARCH_X86_64
+SECTION_IBT_SHSTK

x265-4.1.tar/source/common/x86/pixeladd8.asm -> x265-4.2.tar/source/common/x86/pixeladd8.asm Changed

x265-4.1.tar/source/common/x86/sad-a.asm -> x265-4.2.tar/source/common/x86/sad-a.asm Changed

x265-4.1.tar/source/common/x86/sad16-a.asm -> x265-4.2.tar/source/common/x86/sad16-a.asm Changed

x265-4.1.tar/source/common/x86/seaintegral.asm -> x265-4.2.tar/source/common/x86/seaintegral.asm Changed

x265-4.1.tar/source/common/x86/ssd-a.asm -> x265-4.2.tar/source/common/x86/ssd-a.asm Changed

x265-4.1.tar/source/common/x86/v4-ipfilter16.asm -> x265-4.2.tar/source/common/x86/v4-ipfilter16.asm Changed

x265-4.1.tar/source/common/x86/v4-ipfilter8.asm -> x265-4.2.tar/source/common/x86/v4-ipfilter8.asm Changed

x265-4.1.tar/source/common/x86/x86inc.asm -> x265-4.2.tar/source/common/x86/x86inc.asm Changed

@@ -65,6 +65,9 @@
         %define UNIX64 1
     %endif
 %endif
+%ifndef ENABLE_CET
+%define ENABLE_CET 0
+%endif
 
 %define FORMAT_ELF 0
 %ifidn __OUTPUT_FORMAT__,elf
@@ -90,7 +93,30 @@
         SECTION .rodata align=%1
     %endif
 %endmacro
-
+%macro SECTION_IBT_SHSTK 0
+  %if ENABLE_CET
+    %ifidn __OUTPUT_FORMAT__,win32
+    %elif WIN64
+    %else
+         SECTION .note.gnu.property note
+         align 8
+         dd    .x1 - .x0      ; data size for "GNU\0"
+         dd    .x4 - .x1      ; Elf_Prop size
+         dd    5              ; ELF::NT_GNU_PROPERTY_TYPE_0
+    .x0:
+         db    "GNU", 0
+    .x1:
+         align 8
+         dd    0xc0000002     ; ELF::GNU_PROPERTY_X86_FEATURE_1_AND
+         dd    .x3 - .x2      ; data size
+    .x2:
+         dd    0x3            ; ELF::GNU_PROPERTY_X86_FEATURE_1_SHSTK | ELF::GNU_PROPERTY_X86_FEATURE_1_IBT
+    .x3:
+         align 8
+    .x4:
+    %endif
+  %endif
+%endmacro
 %if WIN64
     %define PIC
 %elif ARCH_X86_64 == 0
@@ -737,6 +763,13 @@
     %assign stack_size 0        ; amount of stack space that can be freely used inside a function
     %assign stack_size_padded 0 ; total amount of allocated stack space, including space for callee-saved xmm registers on WIN64 and alignment padding
     %assign xmm_regs_used 0     ; number of XMM registers requested, used for dealing with callee-saved registers on WIN64 and vzeroupper
+    %if ENABLE_CET
+      %if ARCH_X86_64
+        endbr64
+      %else
+        endbr32
+      %endif
+    %endif
     %ifnidn %3, ""
         PROLOGUE %3
     %endif

x265-4.1.tar/source/common/yuv.cpp -> x265-4.2.tar/source/common/yuv.cpp Changed

x265-4.1.tar/source/dynamicHDR10/CMakeLists.txt -> x265-4.2.tar/source/dynamicHDR10/CMakeLists.txt Changed

x265-4.1.tar/source/encoder/CMakeLists.txt -> x265-4.2.tar/source/encoder/CMakeLists.txt Changed

x265-4.1.tar/source/encoder/analysis.cpp -> x265-4.2.tar/source/encoder/analysis.cpp Changed

@@ -72,14 +72,37 @@
 
 Analysis::Analysis()
 {
+    m_bTryLossless = false;
+    m_bChromaSa8d = false;
+    m_bHD = false;
+
+    memset(m_modeFlag, 0, sizeof(m_modeFlag));
+    memset(m_checkMergeAndSkipOnly, 0, sizeof(m_checkMergeAndSkipOnly));
+
+    for (int i = 0; i < NUM_CU_DEPTH; i++)
+    {
+        m_modeDepthi.bestMode = NULL;
+        memset(m_modeDepthi.pred, 0, sizeof(m_modeDepthi.pred));
+    }
+
     m_reuseInterDataCTU = NULL;
     m_reuseRef = NULL;
-    m_bHD = false;
-    m_modeFlag0 = false;
-    m_modeFlag1 = false;
-    m_checkMergeAndSkipOnly0 = false;
-    m_checkMergeAndSkipOnly1 = false;
+    m_reuseDepth = NULL;
+    m_reuseModes = NULL;
+    m_reusePartSize = NULL;
+    m_reuseMergeFlag = NULL;
+    m_reuseMv0 = NULL;
+    m_reuseMv1 = NULL;
+    m_reuseMvpIdx0 = NULL;
+    m_reuseMvpIdx1 = NULL;
+    cacheCost = NULL;
+    m_additionalCtuInfo = NULL;
+    m_prevCtuInfoChange = NULL;
+
     m_evaluateInter = 0;
+    m_refineLevel = 0;
+
+    memset(m_splitRefIdx, 0, sizeof(m_splitRefIdx));
 }
 
 bool Analysis::create(ThreadLocalData *tld)
@@ -135,6 +158,153 @@
     X265_FREE(cacheCost);
 }
 
+void Analysis::computeMVForPUs(CUData& ctu, const CUGeom& cuGeom, int qp, Frame& frame)
+{
+    int areaId = 0;
+    int finalIdx = 0;
+
+    uint32_t depth = cuGeom.depth;
+    uint32_t nextDepth = depth + 1;
+
+    uint32_t cuSize = 1 << cuGeom.log2CUSize;
+    bool mightSplit = (cuSize > m_param->minCUSize);
+
+    uint32_t cuX = ctu.m_cuPelX + g_zscanToPelXcuGeom.absPartIdx;
+    uint32_t cuY = ctu.m_cuPelY + g_zscanToPelYcuGeom.absPartIdx;
+
+    if (cuSize != m_param->maxCUSize)
+    {
+        uint32_t subCUSize = m_param->maxCUSize / 2;
+        areaId = (cuX >= subCUSize) + 2 * (cuY >= subCUSize) + 1;
+    }
+
+    if (mightSplit)
+    {
+        int nextQP = qp;
+        for (uint32_t subPartIdx = 0; subPartIdx < 4; subPartIdx++)
+        {
+            const CUGeom& childGeom = *(&cuGeom + cuGeom.childOffset + subPartIdx);
+            if (m_slice->m_pps->bUseDQP && nextDepth <= m_slice->m_pps->maxCuDQPDepth)
+                nextQP = setLambdaFromQP(ctu, calculateQpforCuSize(ctu, childGeom));
+
+            computeMVForPUs(ctu, childGeom, nextQP, frame);
+        }
+    }
+
+    ModeDepth& md = m_modeDepthcuGeom.depth;
+    CUData& cu = md.predPRED_2Nx2N.cu;
+
+    for (int i = 0; i < MAX_NUM_PU_SIZES; i++)
+    {
+        const PUBlock& pu = g_puLookupi;
+        int startIdx = g_puStartIdxpu.width + pu.heightstatic_cast<int>(pu.partsize);
+
+        if (pu.width > cuSize || pu.height > cuSize || (pu.width != cuSize && pu.height != cuSize))
+            continue;
+
+        if (!m_param->bEnableAMP && pu.isAmp)
+            continue;
+        if (!m_param->bEnableRectInter && pu.width != pu.height && !pu.isAmp)
+            continue;
+
+        int blockWidth = pu.isAmp ? X265_MAX(pu.width, pu.height) : pu.width;
+        int blockHeight = pu.isAmp ? blockWidth : pu.height;
+
+        int numColsCTU = m_param->maxCUSize / blockWidth;
+        int numRowsCTU = m_param->maxCUSize / blockHeight;
+
+        int puOffset = 0;
+        if (pu.isAmp)
+            puOffset = numRowsCTU * numColsCTU;
+        else if (pu.partsize == SIZE_2NxN)
+            puOffset = numColsCTU;
+        else if (pu.partsize == SIZE_Nx2N)
+            puOffset = 1;
+
+        int col = (cuX - ctu.m_cuPelX) / blockWidth;
+        int row = (cuY - ctu.m_cuPelY) / blockHeight;
+
+        finalIdx = startIdx + row * numColsCTU + col;
+
+        int subIdx =finalIdx - startIdx;
+
+        int puRow = subIdx / numColsCTU;
+        int puCol = subIdx % numColsCTU;
+ 
+        int leftIdx = (puCol > 0) ? startIdx + puRow * numColsCTU + (puCol - 1) : -1;
+        int aboveIdx = (puRow > 0) ? startIdx + (puRow - 1) * numColsCTU + puCol : -1;
+        int aboveLeftIdx = (puRow > 0 && puCol > 0) ? startIdx + (puRow - 1) * numColsCTU + (puCol - 1) : -1;
+        int aboveRightIdx = (puRow > 0 && puCol < numColsCTU - 1) ? startIdx + (puRow - 1) * numColsCTU + (puCol + 1) : -1;
+
+        int neighborIdxMD_ABOVE_LEFT + 1 = { leftIdx, aboveIdx, aboveRightIdx, -1, aboveLeftIdx};
+
+        cu.initSubCU(ctu, cuGeom, qp);
+        cu.setPartSizeSubParts(pu.partsize);
+        setLambdaFromQP(cu, qp);
+        puMotionEstimation(m_slice, cuGeom, cu, m_frame->m_fencPic, puOffset, pu.partsize, areaId, finalIdx, false, neighborIdx);
+    }
+}
+
+void Analysis::deriveMVsForCTU(CUData& ctu, const CUGeom& cuGeom, Frame& frame)
+{
+    m_slice = ctu.m_slice;
+    m_frame = &frame;
+    m_param = m_frame->m_param;
+
+    ModeDepth& md = m_modeDepth0;
+
+    int numPredDir = m_slice->isInterP() ? 1 : 2;
+
+    // Full CTU
+    int baseQP = setLambdaFromQP(ctu, ctu.m_slice->m_pps->bUseDQP ? calculateQpforCuSize(ctu, cuGeom) : ctu.m_slice->m_sliceQp);
+
+    md.predPRED_2Nx2N.cu.initSubCU(ctu, cuGeom, baseQP);
+    md.predPRED_2Nx2N.cu.setPartSizeSubParts(SIZE_2Nx2N);
+
+    puMotionEstimation(m_slice, cuGeom, md.predPRED_2Nx2N.cu, frame.m_fencPic, 0, SIZE_2Nx2N, 0, 0, true);
+
+    // Sub-CUs
+    if (m_param->maxCUSize != m_param->minCUSize)
+    {
+        for (int sub = 0; sub < 4; sub++)
+        {
+            ModeDepth& md1 = m_modeDepth1;
+
+            const CUGeom& childGeom = *(&cuGeom + cuGeom.childOffset + sub);
+            int qp = setLambdaFromQP(ctu, ctu.m_slice->m_pps->bUseDQP ? calculateQpforCuSize(ctu, childGeom) : ctu.m_slice->m_sliceQp);
+
+            md1.predPRED_2Nx2N.cu.initSubCU(ctu, childGeom, qp);
+            md1.predPRED_2Nx2N.cu.setPartSizeSubParts(SIZE_2Nx2N);
+
+            puMotionEstimation(m_slice, childGeom, md1.predPRED_2Nx2N.cu, frame.m_fencPic, 0, SIZE_2Nx2N, sub + 1, 0, true);
+        }
+    }
+
+    const Frame* colPic = m_slice->m_refFrameListm_slice->isInterB() && !m_slice->m_colFromL0Flagm_slice->m_colRefIdx;
+    const CUData* colCU = colPic->m_encData->getPicCTU(ctu.m_cuAddr);
+
+    for (int list = 0; list < numPredDir; list++)
+    {
+        int numRef = ctu.m_slice->m_numRefIdxlist;
+
+        for (int ref = 0; ref < numRef; ref++)
+        {
+            MV medianMv;
+            bool valid = ctu.getMedianColMV(colCU, colPic, list, ref, medianMv);
+            if (!valid)
+                continue;
+
+            for (int areaIdx = 0; areaIdx < 5; areaIdx++)
+            {
+                m_areaBestMVareaIdxlistref = medianMv;
+            }
+        }
+    }
+
+    computeMVForPUs(ctu, cuGeom, baseQP, frame);
+
+}
+
 Mode& Analysis::compressCTU(CUData& ctu, Frame& frame, const CUGeom& cuGeom, const Entropy& initialContext)
 {
     m_slice = ctu.m_slice;
@@ -2960,12 +3130,19 @@
 
         if (m_refineLevel > 1 || (m_refineLevel && parentCTU.m_predModecuGeom.absPartIdx == MODE_SKIP  && !mode.cu.isSkipped(0)))
         {
-            if (parentCTU.m_cuDepthcuGeom.absPartIdx < 4 && mightNotSplit)
-                m_evaluateInter = 1;
+            if ((m_slice->m_origSliceType != I_SLICE))
+            {
+                if (parentCTU.m_cuDepthcuGeom.absPartIdx < 4 && mightNotSplit)
+                    m_evaluateInter = 1;
+                else
+                    bDecidedDepth = true;
+                m_param->rdLevel > 4 ? compressInterCU_rd5_6(parentCTU, cuGeom, qp) : compressInterCU_rd0_4(parentCTU, cuGeom, qp);
+                m_evaluateInter = 0;
+            }
             else
-                bDecidedDepth = true;
-            m_param->rdLevel > 4 ? compressInterCU_rd5_6(parentCTU, cuGeom, qp) : compressInterCU_rd0_4(parentCTU, cuGeom, qp);
-            m_evaluateInter = 0;
+            {
+                compressIntraCU(parentCTU, cuGeom, qp);
+            }
         }
     }
     if (!bDecidedDepth || split)
@@ -3165,12 +3342,18 @@
             if (m_param->maxSlices > 1)
             {
                 // NOTE: First row in slice can't negative
-                if (X265_MIN(candMvFieldi0.mv.y, candMvFieldi1.mv.y) < m_sliceMinY)
+                if (candMvFieldi0.refIdx >= 0 && candMvFieldi0.mv.notZero() && candMvFieldi0.mv.y < m_sliceMinY)
+                    continue;
+
+                if (candMvFieldi1.refIdx >= 0 && candMvFieldi1.mv.notZero() && candMvFieldi1.mv.y < m_sliceMinY)
                     continue;
 
                 // Last row in slice can't reference beyond bound since it is another slice area
                 // TODO: we may beyond bound in future since these area have a chance to finish because we use parallel slices. Necessary prepare research on load balance
-                if (X265_MAX(candMvFieldi0.mv.y, candMvFieldi1.mv.y) > m_sliceMaxY)
+                if (candMvFieldi0.refIdx >= 0 && candMvFieldi0.mv.notZero() && candMvFieldi0.mv.y > m_sliceMaxY)
+                    continue;
+
+                if (candMvFieldi1.refIdx >= 0 && candMvFieldi1.mv.notZero() && candMvFieldi1.mv.y > m_sliceMaxY)
                     continue;
             }
 
@@ -3299,12 +3482,18 @@
             if (m_param->maxSlices > 1)
             {
                 // NOTE: First row in slice can't negative
-                if (X265_MIN(candMvFieldi0.mv.y, candMvFieldi1.mv.y) < m_sliceMinY)
+                if (candMvFieldi0.refIdx >= 0 && candMvFieldi0.mv.notZero() && candMvFieldi0.mv.y < m_sliceMinY)
+                    continue;
+
+                if (candMvFieldi1.refIdx >= 0 && candMvFieldi1.mv.notZero() && candMvFieldi1.mv.y < m_sliceMinY)
                     continue;
 
                 // Last row in slice can't reference beyond bound since it is another slice area
                 // TODO: we may beyond bound in future since these area have a chance to finish because we use parallel slices. Necessary prepare research on load balance
-                if (X265_MAX(candMvFieldi0.mv.y, candMvFieldi1.mv.y) > m_sliceMaxY)
+                if (candMvFieldi0.refIdx >= 0 && candMvFieldi0.mv.notZero() && candMvFieldi0.mv.y > m_sliceMaxY)
+                    continue;
+
+                if (candMvFieldi1.refIdx >= 0 && candMvFieldi1.mv.notZero() && candMvFieldi1.mv.y > m_sliceMaxY)
                     continue;
             }

x265-4.1.tar/source/encoder/analysis.h -> x265-4.2.tar/source/encoder/analysis.h Changed

@@ -130,6 +130,24 @@
     Mode& compressCTU(CUData& ctu, Frame& frame, const CUGeom& cuGeom, const Entropy& initialContext);
     int32_t loadTUDepth(CUGeom cuGeom, CUData parentCTU);
 
+    /**
+     * @brief Build CTU-level and area-level MVP seeds used by threaded ME.
+     *
+     * Performs an initial 2Nx2N search on the full CTU (and first split depth
+     * when available), propagates temporal/colocated medians, and then drives
+     * per-PU motion estimation.
+     */
+    void deriveMVsForCTU(CUData& ctu, const CUGeom& cuGeom, Frame& frame);
+
+    /**
+     * @brief Recursively walk CU partitions and run ME for each enabled PU shape.
+     *
+     * Computes the PU index mapping (`finalIdx`) used by CTU MV storage and
+     * submits each PU to puMotionEstimation() with neighbor indices for MVP
+     * derivation.
+     */
+    void computeMVForPUs(CUData& ctu, const CUGeom& cuGeom, int qp, Frame& frame);
+
 protected:
     /* Analysis data for save/load mode, writes/reads data based on absPartIdx */
     x265_analysis_inter_data*  m_reuseInterDataCTU;

x265-4.1.tar/source/encoder/api.cpp -> x265-4.2.tar/source/encoder/api.cpp Changed

@@ -309,7 +309,7 @@
 {
     if (!enc || !param_in)
         return -1;
-    x265_param save;
+    x265_param save = {};
     Encoder* encoder = static_cast<Encoder*>(enc);
     if (strlen(encoder->m_param->csvfn) && param_in->csvfpt != NULL)
          encoder->m_param->csvfpt = param_in->csvfpt;
@@ -965,8 +965,8 @@
 
         if (maxReuseLevel > 4)
         {
-            X265_FREE((analysis->interData)->mergeFlag);
-            X265_FREE((analysis->interData)->partSize);
+             X265_FREE((analysis->interData)->mergeFlag);
+             X265_FREE((analysis->interData)->partSize);
         }
         if (maxReuseLevel >= 7)
         {
@@ -1403,6 +1403,8 @@
                     /* detailed performance statistics */
                     fprintf(csvfp, ", DecideWait (ms), Row0Wait (ms), Wall time (ms), Ref Wait Wall (ms), Total CTU time (ms),"
                         "Stall Time (ms), Total frame time (ms), Avg WPP, Row Blocks");
+
+                    fprintf(csvfp, ", Total ThreadedME Wait Time (ms), Total ThreadedME Time (ms)");
 #if ENABLE_LIBVMAF
                     fprintf(csvfp, ", VMAF Frame Score");
 #endif
@@ -1539,6 +1541,9 @@
                                                                                      frameStats->totalFrameTime);
 
         fprintf(param->csvfpt, " %.3lf, %d", frameStats->avgWPP, frameStats->countRowBlocks);
+
+        fprintf(param->csvfpt, ", %.1lf, %.1lf", frameStats->tmeWaitTime / 1000.0, frameStats->tmeTime / 1000.0);
+
 #if ENABLE_LIBVMAF
         fprintf(param->csvfpt, ", %lf", frameStats->vmafFrameScore);
 #endif

x265-4.1.tar/source/encoder/dpb.cpp -> x265-4.2.tar/source/encoder/dpb.cpp Changed

x265-4.1.tar/source/encoder/encoder.cpp -> x265-4.2.tar/source/encoder/encoder.cpp Changed

@@ -39,6 +39,7 @@
 #include "ratecontrol.h"
 #include "dpb.h"
 #include "nal.h"
+#include "threadedme.h"
 
 #include "x265.h"
 
@@ -70,7 +71,7 @@
 
 DolbyVisionProfileSpec dovi =
 {
-    { 1, 1, 1, 1, 1, 5, 1,  2, 2, 2, 50 },
+    { 1, 1, 1, 1, 1, 5, 1, 16, 9, 15, 50 },
     { 1, 1, 1, 1, 1, 5, 0, 16, 9, 9, 81 },
     { 1, 1, 1, 1, 1, 5, 0,  1, 1, 1, 82 },
     { 1, 1, 1, 1, 1, 5, 0, 18, 9, 9, 84 }
@@ -132,6 +133,7 @@
     m_numLumaWPBiFrames = 0;
     m_numChromaWPBiFrames = 0;
     m_lookahead = NULL;
+    m_threadedME = NULL;
     m_rateControl = NULL;
     m_dpb = NULL;
     m_numDelayedPic = 0;
@@ -254,10 +256,16 @@
         p->bEnableWavefront = 0;
     }
 
+    // For zero-latency tune, frameNumThreads must be set to 1
+    if (p->tune && (!strcmp(p->tune, "zerolatency") || !strcmp(p->tune, "zero-latency")))
+    {
+        p->frameNumThreads = 1;
+    }
+
     bool allowPools = !strlen(p->numaPools) || strcmp(p->numaPools, "none");
 
     // Trim the thread pool if --wpp, --pme, and --pmode are disabled
-    if (!p->bEnableWavefront && !p->bDistributeModeAnalysis && !p->bDistributeMotionEstimation && !p->lookaheadSlices)
+    if (!p->bEnableWavefront && !p->bDistributeModeAnalysis && !p->bDistributeMotionEstimation && !p->lookaheadSlices && !p->bThreadedME)
         allowPools = false;
 
     m_numPools = 0;
@@ -269,7 +277,7 @@
         {
             // auto-detect frame threads
             int cpuCount = ThreadPool::getCpuCount();
-            ThreadPool::getFrameThreadsCount(p, cpuCount);
+            p->frameNumThreads = ThreadPool::getFrameThreadsCount(p, cpuCount);
         }
     }
 
@@ -284,9 +292,12 @@
             x265_log(p, X265_LOG_WARNING, "No thread pool allocated, --pmode disabled\n");
         if (p->lookaheadSlices)
             x265_log(p, X265_LOG_WARNING, "No thread pool allocated, --lookahead-slices disabled\n");
+        if (p->bThreadedME)
+            x265_log(p, X265_LOG_WARNING, "No thread pool allocated, --threaded-me disabled\n");
 
         // disable all pool features if the thread pool is disabled or unusable.
         p->bEnableWavefront = p->bDistributeModeAnalysis = p->bDistributeMotionEstimation = p->lookaheadSlices = 0;
+        p->bThreadedME = 0;
     }
 
     x265_log(p, X265_LOG_INFO, "Slices                              : %d\n", p->maxSlices);
@@ -299,6 +310,8 @@
         len += snprintf(buf + len,  sizeof(buf) - len, "%spmode", len ? "+" : "");
     if (p->bDistributeMotionEstimation)
         len += snprintf(buf + len, sizeof(buf) - len, "%spme ", len ? "+" : "");
+    if (p->bThreadedME)
+        len += snprintf(buf + len, sizeof(buf) - len, "%sthreaded-me", len ? "+": "");
     if (!len)
         strcpy(buf, "none");
 
@@ -310,17 +323,37 @@
         m_frameEncoderi->m_nalList.m_annexB = !!m_param->bAnnexB;
     }
 
+    if (p->bThreadedME)
+    {
+        m_threadedME = new ThreadedME(m_param, *this);
+    }
+
     if (m_numPools)
     {
+        // First threadpool belongs to ThreadedME, if the feature is enabled
+        if (p->bThreadedME)
+        {
+            m_threadedME->m_pool = &m_threadPool0;
+            m_threadedME->m_jpId = 0;
+
+            m_threadPool0.m_numProviders = 1;
+            m_threadPool0.m_jpTablem_threadedME->m_jpId = m_threadedME;
+        }
+
+        int numFrameThreadPools = (!m_param->bThreadedME) ? m_numPools : m_numPools - 1;
+
         for (int i = 0; i < m_param->frameNumThreads; i++)
         {
-            int pool = i % m_numPools;
+            // Since first pool belongs to ThreadedME
+            int pool = static_cast<int>(p->bThreadedME) + i % numFrameThreadPools;
             m_frameEncoderi->m_pool = &m_threadPoolpool;
             m_frameEncoderi->m_jpId = m_threadPoolpool.m_numProviders++;
             m_threadPoolpool.m_jpTablem_frameEncoderi->m_jpId = m_frameEncoderi;
         }
-        for (int i = 0; i < m_numPools; i++)
-            m_threadPooli.start();
+
+
+        for (int j = 0; j < m_numPools; j++)
+            m_threadPoolj.start();
     }
     else
     {
@@ -348,7 +381,7 @@
         lookAheadThreadPool = ThreadPool::allocThreadPools(p, pools, 1);
     }
     else
-        lookAheadThreadPool = m_threadPool;
+        lookAheadThreadPool = (!m_param->bThreadedME) ? m_threadPool : &m_threadPool1;
     m_lookahead = new Lookahead(m_param, lookAheadThreadPool);
     if (pools)
     {
@@ -361,6 +394,22 @@
     m_lookahead->m_numPools = pools;
     m_dpb = new DPB(m_param);
 
+    if (p->bThreadedME)
+    {
+        if (!m_threadedME->create())
+        {
+            m_param->bThreadedME = 0;
+            X265_FREE(m_threadedME);
+            m_threadedME = NULL;
+
+            x265_log(m_param, X265_LOG_ERROR, "Failed to create threadedME thread pool, --threaded-me disabled");
+        }
+        else
+        {
+            m_threadedME->start();
+        }
+    }
+
     m_rateControl = new RateControl(*m_param, this);
     if (!m_param->bResetZoneConfig)
     {
@@ -474,6 +523,7 @@
         m_aborted = true;
 
     initRefIdx();
+
     if (strlen(m_param->analysisSave) && m_param->bUseAnalysisFile)
     {
         char* temp = strcatFilename(m_param->analysisSave, ".temp");
@@ -584,7 +634,10 @@
 
     if (m_lookahead)
         m_lookahead->stopJobs();
-    
+
+    if (m_threadedME)
+        m_threadedME->stopJobs();
+
     for (int i = 0; i < m_param->frameNumThreads; i++)
     {
         if (m_frameEncoderi)
@@ -927,6 +980,12 @@
         delete m_lookahead;
     }
 
+    if (m_threadedME)
+    {
+        m_threadedME->destroy();
+        delete m_threadedME;
+    }
+
     delete m_dpb;
     if (!m_param->bResetZoneConfig && m_param->rc.zonefileCount)
     {
@@ -1082,6 +1141,16 @@
     }
 
     int numPayloads = pic_in->userSEI.numPayloads + toneMapPayload + userPayload;
+
+    // TODO: we may reuse buffer if become smaller than exist buffer
+    if (frame->m_userSEI.payloads && numPayloads != frame->m_userSEI.numPayloads)
+    {
+        for (int i = 0; i < frame->m_userSEI.numPayloads; i++)
+            delete frame->m_userSEI.payloadsi.payload;
+        delete frame->m_userSEI.payloads;
+        frame->m_userSEI.payloads = NULL;
+    }
+
     frame->m_userSEI.numPayloads = numPayloads;
 
     if (frame->m_userSEI.numPayloads)
@@ -1102,6 +1171,12 @@
             else
                 input = pic_in->userSEI.payloadsi;
 
+            // TODO: condition may improve, because buffer size may change from big to small, but never back to original allocate size
+            if (frame->m_userSEI.payloadsi.payload && frame->m_userSEI.payloadsi.payloadSize < input.payloadSize)
+            {
+                delete frame->m_userSEI.payloadsi.payload;
+                frame->m_userSEI.payloadsi.payload = NULL;
+            }
             if (!frame->m_userSEI.payloadsi.payload)
                 frame->m_userSEI.payloadsi.payload = new uint8_tinput.payloadSize;
             memcpy(frame->m_userSEI.payloadsi.payload, input.payload, input.payloadSize);
@@ -1538,10 +1613,20 @@
                     {
                         inFramelayer->m_fencPic->m_cuOffsetY = m_sps.cuOffsetY;
                         inFramelayer->m_fencPic->m_buOffsetY = m_sps.buOffsetY;
+                        if (m_param->bEnableTemporalFilter)
+                        {
+                            inFramelayer->m_mcstffencPic->m_cuOffsetY = m_sps.cuOffsetY;
+                            inFramelayer->m_mcstffencPic->m_buOffsetY = m_sps.buOffsetY;
+                        }
                         if (m_param->internalCsp != X265_CSP_I400)
                         {
                             inFramelayer->m_fencPic->m_cuOffsetC = m_sps.cuOffsetC;
                             inFramelayer->m_fencPic->m_buOffsetC = m_sps.buOffsetC;
+                            if (m_param->bEnableTemporalFilter)
+                            {
+                                inFramelayer->m_mcstffencPic->m_cuOffsetC = m_sps.cuOffsetC;
+                                inFramelayer->m_mcstffencPic->m_buOffsetC = m_sps.buOffsetC;
+                            }
                         }
                     }
                     else
@@ -1558,12 +1643,20 @@
                         {
                             m_sps.cuOffsetY = inFramelayer->m_fencPic->m_cuOffsetY;
                             m_sps.buOffsetY = inFramelayer->m_fencPic->m_buOffsetY;
+                            if (m_param->bEnableTemporalFilter)
+                            {
+                                inFramelayer->m_mcstffencPic->m_cuOffsetY = m_sps.cuOffsetY;
+                                inFramelayer->m_mcstffencPic->m_buOffsetY = m_sps.buOffsetY;
+                            }
                             if (m_param->internalCsp != X265_CSP_I400)
                             {
                                 m_sps.cuOffsetC = inFramelayer->m_fencPic->m_cuOffsetC;
-                                m_sps.cuOffsetY = inFramelayer->m_fencPic->m_cuOffsetY;
                                 m_sps.buOffsetC = inFramelayer->m_fencPic->m_buOffsetC;
-                                m_sps.buOffsetY = inFramelayer->m_fencPic->m_buOffsetY;
+                                if (m_param->bEnableTemporalFilter)
+                                {
+                                    inFramelayer->m_mcstffencPic->m_cuOffsetC = m_sps.cuOffsetC;
+                                    inFramelayer->m_mcstffencPic->m_buOffsetC = m_sps.buOffsetC;
+                                }
                             }
                         }
                     }
@@ -1811,7 +1904,7 @@
             if (!m_pocLast)
             {
                 /*One shot allocation of frames in OriginalPictureBuffer*/
-                int numFramesinOPB = X265_MAX(m_param->bframes, (inFrame0->m_mcstf->m_range << 1)) + 1;
+                int numFramesinOPB = X265_MAX(m_param->bframes, m_param->frameNumThreads) + 2;
                 for (int i = 0; i < numFramesinOPB; i++)
                 {
                     Frame* dupFrame = new Frame;
@@ -2715,14 +2808,14 @@
 
     if (m_param->bEnablePsnr)
     {
-        len += snprintf(buffer + len, sizeof(buffer) - len,"  PSNR Mean: Y:%.3lf U:%.3lf V:%.3lf",
+        len += snprintf(buffer + len, bufferSize - len,"  PSNR Mean: Y:%.3lf U:%.3lf V:%.3lf",
                        stat.m_psnrSumY / (double)stat.m_numPics,
                        stat.m_psnrSumU / (double)stat.m_numPics,
                        stat.m_psnrSumV / (double)stat.m_numPics);
     }
     if (m_param->bEnableSsim)
     {
-        snprintf(buffer + len, sizeof(buffer) - len, "  SSIM Mean: %.6lf (%.3lfdB)",
+        snprintf(buffer + len, bufferSize - len, "  SSIM Mean: %.6lf (%.3lfdB)",
                 stat.m_globalSsim / (double)stat.m_numPics,
                 x265_ssim2dB(stat.m_globalSsim / (double)stat.m_numPics));
     }
@@ -2803,6 +2896,12 @@
         for (int i = 0; i < m_param->frameNumThreads; i++)
             cuStats.accumulate(m_frameEncoderi->m_cuStats, *m_param);
 
+        if (m_param->bThreadedME)
+        {
+            m_threadedME->collectStats();
+            cuStats.accumulate(m_threadedME->m_cuStats, *m_param);
+        }
+
         if (!cuStats.totalCTUTime)
             return;
 
@@ -2817,7 +2916,7 @@
             batchElapsedTime + coopSliceElapsedTime;
 
         int64_t totalWorkerTime = cuStats.totalCTUTime + cuStats.loopFilterElapsedTime + cuStats.pmodeTime +
-            cuStats.pmeTime + lookaheadWorkerTime + cuStats.weightAnalyzeTime;
+            cuStats.pmeTime + lookaheadWorkerTime + cuStats.weightAnalyzeTime + cuStats.tmeTime;
         int64_t elapsedEncodeTime = x265_mdate() - m_encodeStartTime;
 
         int64_t interRDOTotalTime = 0, intraRDOTotalTime = 0;
@@ -2849,6 +2948,12 @@
                 (double)cuStats.countPMETasks / cuStats.countPMEMasters,
                 ELAPSED_MSEC(cuStats.pmeTime) / cuStats.countPMETasks);
         }
+        else if (m_param->bThreadedME && cuStats.countTmeTasks)
+        {
+            x265_log(m_param, X265_LOG_INFO, "CU: %%%05.2lf time spent in motion estimation, averaging %.3lf CU inter modes per CTU\n",
+                100.0 * (cuStats.motionEstimationElapsedTime + cuStats.tmeTime) / totalWorkerTime,
+                (double)cuStats.countMotionEstimate / cuStats.totalCTUs);
+        }
         else
         {
             x265_log(m_param, X265_LOG_INFO, "CU: %%%05.2lf time spent in motion estimation, averaging %.3lf CU inter modes per CTU\n",
@@ -2936,6 +3041,11 @@
             ELAPSED_SEC(totalWorkerTime),
             cuStats.totalCTUs / ELAPSED_SEC(totalWorkerTime));
 
+        if (m_param->bThreadedME && cuStats.countTmeBlockedCTUs)
+            x265_log(m_param, X265_LOG_INFO, "CU: " X265_LL " CTUs blocked by ThreadedME, %%%05.2lf of total CTUs\n",
+                cuStats.countTmeBlockedCTUs,
+                100.0 * cuStats.countTmeBlockedCTUs / cuStats.totalCTUs);
+
         if (m_threadPool)
             x265_log(m_param, X265_LOG_INFO, "CU: %.3lf average worker utilization, %%%05.2lf of theoretical maximum utilization\n",
                 (double)totalWorkerTime / elapsedEncodeTime,
@@ -3135,6 +3245,10 @@
             frameStats->totalCTUTime = ELAPSED_MSEC(0, curEncoder->m_totalWorkerElapsedTimelayer);
             frameStats->stallTime = ELAPSED_MSEC(0, curEncoder->m_totalNoWorkerTimelayer);
             frameStats->totalFrameTime = ELAPSED_MSEC(curFrame->m_encodeStartTime, x265_mdate());
+
+            frameStats->tmeTime = curEncoder->m_totalThreadedMETimelayer;
+            frameStats->tmeWaitTime = curEncoder->m_totalThreadedMEWaitlayer;
+
             if (curEncoder->m_totalActiveWorkerCount)
                 frameStats->avgWPP = (double)curEncoder->m_totalActiveWorkerCount / curEncoder->m_activeWorkerCountSamples;
             else
@@ -3723,6 +3837,7 @@
             p->rc.qp = zone->rc.qp;
             p->rc.aqMode = X265_AQ_NONE;
             p->rc.hevcAq = 0;
+            p->bAQMotion = 0;
         }
         if (p->rc.aqMode == 0 && p->rc.cuTree)
         {
@@ -3938,6 +4053,7 @@
         p->rc.bitrate = 0;
         p->rc.cuTree = 0;
         p->rc.aqStrength = 0;
+        p->bAQMotion = 0;
     }
 
     if (p->rc.aqMode == 0 && p->rc.cuTree)

x265-4.1.tar/source/encoder/encoder.h -> x265-4.2.tar/source/encoder/encoder.h Changed

x265-4.1.tar/source/encoder/entropy.cpp -> x265-4.2.tar/source/encoder/entropy.cpp Changed

@@ -1782,7 +1782,7 @@
                 else
 #endif
                     WRITE_FLAG(!!wp0.wtPresent, "luma_weight_lX_flag");
-                totalSignalledWeightFlags += wp0.wtPresent;
+                totalSignalledWeightFlags = totalSignalledWeightFlags + wp0.wtPresent;
             }
 
             if (bChroma)
@@ -1796,7 +1796,7 @@
                     else
 #endif
                         WRITE_FLAG(!!wp1.wtPresent, "chroma_weight_lX_flag");
-                    totalSignalledWeightFlags += 2 * wp1.wtPresent;
+                    totalSignalledWeightFlags = totalSignalledWeightFlags + 2 * wp1.wtPresent;
                 }
             }
 
@@ -1893,7 +1893,7 @@
         codeNumber = (codeNumber >> absGoRice) - COEF_REMAIN_BIN_REDUCTION;
         {
             unsigned long idx;
-            CLZ(idx, codeNumber + 1);
+            BSR(idx, codeNumber + 1);
             length = idx;
             X265_CHECK((codeNumber != 0) || (length == 0), "length check failure\n");
             codeNumber -= (1 << idx) - 1;
@@ -2206,7 +2206,7 @@
             {
                 {
                     unsigned long cidx;
-                    CLZ(cidx, codeNumber + 1);
+                    BSR(cidx, codeNumber + 1);
                     length = cidx;
                 }
                 X265_CHECK((codeNumber != 0) || (length == 0), "length check failure\n");
@@ -2319,8 +2319,8 @@
     int scanPosSigOff = scanPosLast - (lastScanSet << MLS_CG_SIZE) - 1;
     ALIGN_VAR_32(uint16_t, absCoeff(1 << MLS_CG_SIZE) + 1);   // extra 2 bytes(+1) space for AVX2 assembly, +1 because (numNonZero<=1) in costCoeffNxN path
     uint32_t numNonZero = 1;
-    unsigned long lastNZPosInCG;
-    unsigned long firstNZPosInCG;
+    unsigned long lastNZPosInCG = 0;
+    unsigned long firstNZPosInCG = 0;
 
 #if _DEBUG
     // Unnecessary, for Valgrind-3.10.0 only
@@ -2410,6 +2410,7 @@
             if (m_bitIf)
             {
                 ALIGN_VAR_32(uint16_t, tmpCoeffSCAN_SET_SIZE);
+                memset(tmpCoeff, 0, sizeof(tmpCoeff));
 
                 // TODO: accelerate by PABSW
                 for (int i = 0; i < MLS_CG_SIZE; i++)
@@ -2488,10 +2489,10 @@
         numNonZero = coeffNumsubSet;
         if (numNonZero > 0)
         {
-            uint32_t idx;
+            uint32_t idx = 0;
             X265_CHECK(subCoeffFlag > 0, "subCoeffFlag is zero\n");
-            CLZ(lastNZPosInCG, subCoeffFlag);
-            CTZ(firstNZPosInCG, subCoeffFlag);
+            BSR(lastNZPosInCG, subCoeffFlag);
+            BSF(firstNZPosInCG, subCoeffFlag);
 
             bool signHidden = (lastNZPosInCG - firstNZPosInCG >= SBH_THRESHOLD);
             const uint8_t ctxSet = (((subSet > 0) + bIsLuma) & 2) + !(c1 & 3);
@@ -2887,7 +2888,7 @@
         // NOTE: lps is non-zero and the maximum of idx is 8 because lps less than 256
         //numBits = g_renormTablelps >> 3;
         unsigned long idx;
-        CLZ(idx, lps);
+        BSR(idx, lps);
         X265_CHECK(state != 63 || idx == 1, "state failure\n");
 
         numBits = 8 - idx;

x265-4.1.tar/source/encoder/frameencoder.cpp -> x265-4.2.tar/source/encoder/frameencoder.cpp Changed

@@ -36,6 +36,8 @@
 #include "nal.h"
 #include "temporalfilter.h"
 
+#include <iostream>
+
 namespace X265_NS {
 void weightAnalyse(Slice& slice, Frame& frame, x265_param& param);
 
@@ -196,10 +198,12 @@
     // 7.4.7.1 - Ceil( Log2( PicSizeInCtbsY ) ) bits
     {
         unsigned long tmp;
-        CLZ(tmp, (numRows * numCols - 1));
+        BSR(tmp, (numRows * numCols - 1));
         m_sliceAddrBits = (uint16_t)(tmp + 1);
     }
 
+    m_tmeDeps.resize(m_numRows);
+
     m_retFrameBuffer = X265_MALLOC(Frame*, m_param->numLayers);
     for (int layer = 0; layer < m_param->numLayers; layer++)
         m_retFrameBufferlayer = NULL;
@@ -374,7 +378,7 @@
 
     ++ucode;
     unsigned long idx;
-    CLZ( idx, ucode );
+    BSR( idx, ucode );
     uint32_t length = (uint32_t)idx * 2 + 1;
 
     return length;
@@ -447,6 +451,8 @@
     m_totalActiveWorkerCount = 0;
     m_activeWorkerCountSamples = 0;
     m_totalWorkerElapsedTimelayer = 0;
+    m_totalThreadedMETimelayer = 0;
+    m_totalThreadedMEWaitlayer = 0;
     m_totalNoWorkerTimelayer = 0;
     m_countRowBlocks = 0;
     m_allRowsAvailableTimelayer = 0;
@@ -915,7 +921,7 @@
      * compressed in a wave-front pattern if WPP is enabled. Row based loop
      * filters runs behind the CTU compression and reconstruction */
 
-    for (uint32_t sliceId = 0; sliceId < m_param->maxSlices; sliceId++)    
+    for (uint32_t sliceId = 0; sliceId < m_param->maxSlices; sliceId++)
         m_rowsm_sliceBaseRowsliceId.active = true;
     
     if (m_param->bEnableWavefront)
@@ -975,8 +981,16 @@
                             m_mreflref.applyWeight(rowIdx, m_numRows, sliceEndRow, sliceId);
                     }
                 }
-
+                
                 enableRowEncoder(m_row_to_idxrow); /* clear external dependency for this row */
+
+                if (m_top->m_threadedME && !slice->isIntra())
+                {
+                    ScopedLock lock(m_tmeDepLock);
+                    m_tmeDepsrow.external = true;
+                    m_top->m_threadedME->enqueueReadyRows(row, layer, this);
+                }
+
                 if (!rowInSlice)
                 {
                     m_row0WaitTimelayer = x265_mdate();
@@ -1022,6 +1036,13 @@
                     }
                 }
 
+                if (m_top->m_threadedME && !slice->isIntra())
+                {
+                    ScopedLock lock(m_tmeDepLock);
+                    m_tmeDepsi.external = true;
+                    m_top->m_threadedME->enqueueReadyRows(i, layer, this);
+                }
+
                 if (!i)
                     m_row0WaitTimelayer = x265_mdate();
                 else if (i == m_numRows - 1)
@@ -1038,6 +1059,11 @@
     vmafFrameLevelScore();
 #endif
 
+    m_tmeDepLock.acquire();
+    m_tmeDeps.clear();
+    m_tmeDeps.resize(m_numRows);
+    m_tmeDepLock.release();
+
     if (m_param->maxSlices > 1)
     {
         PicYuv *reconPic = m_framelayer->m_reconPic0;
@@ -1470,7 +1496,9 @@
     const uint32_t typeNum = m_idx_to_rowrow & 1;
 
     if (!typeNum)
+    {
         processRowEncoder(realRow, m_tldthreadId, layer);
+    }
     else
     {
         m_frameFilter.processRow(realRow, layer);
@@ -1600,6 +1628,12 @@
     if (tld.analysis.m_sliceMaxY < tld.analysis.m_sliceMinY)
         tld.analysis.m_sliceMaxY = tld.analysis.m_sliceMinY = 0;
 
+    if (m_top->m_threadedME && !slice->isIntra())
+    {
+        ScopedLock lock(m_tmeDepLock);
+        m_tmeDepsrow.internal = true;
+        m_top->m_threadedME->enqueueReadyRows(row, layer, this);
+    }
 
     while (curRow.completed < numCols)
     {
@@ -1609,6 +1643,29 @@
         const uint32_t cuAddr = lineStartCUAddr + col;
         CUData* ctu = curEncData.getPicCTU(cuAddr);
         const uint32_t bLastCuInSlice = (bLastRowInSlice & (col == numCols - 1)) ? 1 : 0;
+
+        /* Must wait for TME to finish before initCTU because both threads
+         * operate on the same CUData — the encoder's initCTU would corrupt
+         * data that deriveMVsForCTU is still reading. */
+        if (m_top->m_threadedME && slice->m_sliceType != I_SLICE)
+        {
+            int64_t waitStart = x265_mdate();
+            bool waited = false;
+
+            while (m_framelayer->m_ctuMEFlagscuAddr.get() == 0)
+            {
+#ifdef DETAILED_CU_STATS
+                tld.analysis.m_statsm_jpId.countTmeBlockedCTUs++;
+#endif
+                m_framelayer->m_ctuMEFlagscuAddr.waitForChange(0);
+                waited = true;
+            }
+
+            int64_t waitEnd = x265_mdate();
+            if (waited)
+                ATOMIC_ADD(&m_totalThreadedMEWaitlayer, waitEnd - waitStart);
+        }
+
         ctu->initCTU(*m_framelayer, cuAddr, slice->m_sliceQp, bFirstRowInSlice, bLastRowInSlice, bLastCuInSlice);
 
         if (!layer && bIsVbv)
@@ -2324,88 +2381,179 @@
     }
 }
 
+void compute_film_grain_resolution(int width, int height,
+                                   int& apply_units_resolution_log2,
+                                   int& apply_horz_resolution,
+                                   int& apply_vert_resolution)
+{
+    unsigned long log2_width, log2_height;
+    BSF(log2_width, (unsigned long) width);
+    BSF(log2_height, (unsigned long) height);
+
+    int log2 = (log2_width < log2_height) ? log2_width : log2_height;
+    apply_units_resolution_log2 = log2;
+
+    int unit = 1 << log2;
+    apply_horz_resolution = width / unit;
+    apply_vert_resolution = height / unit;
+
+    return;
+}
+
 void FrameEncoder::readAomModel(AomFilmGrainCharacteristics* m_aomFilmGrain, FILE* Aomfilmgrain)
 {
     char const* errorMessage = "Error reading Aom FilmGrain characteristics\n";
     AomFilmGrain m_afg;
     m_afg.m_chroma_scaling_from_luma = 0;
+    int bitCount = 0;
+    bitCount += 4; // payload_less_than_4byte_flag(1) + film_grain_param_set_idx(3)
     x265_fread((char*)&m_aomFilmGrain->m_apply_grain, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount++;
     x265_fread((char*)&m_aomFilmGrain->m_grain_seed, sizeof(uint16_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=16;
     x265_fread((char*)&m_aomFilmGrain->m_update_grain, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount++;
     x265_fread((char*)&m_aomFilmGrain->m_num_y_points, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=4;
+
     if (m_aomFilmGrain->m_num_y_points)
     {
+        m_aomFilmGrain->point_y_value_increment_bits = 8;
+        bitCount += 3;
+        m_aomFilmGrain->point_y_scaling_bits = 8;
+        bitCount += 2;
         for (int i = 0; i < m_aomFilmGrain->m_num_y_points; i++)
         {
             for (int j = 0; j < 2; j++)
             {
                 x265_fread((char*)&m_aomFilmGrain->m_scaling_points_yij, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+                bitCount+=8;
             }
         }
     }
     x265_fread((char*)&m_aomFilmGrain->m_num_cb_points, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=4;
     if (m_aomFilmGrain->m_num_cb_points)
     {
+        m_aomFilmGrain->point_cb_value_increment_bits = 8;
+        bitCount += 3;
+        m_aomFilmGrain->point_cb_scaling_bits = 8;
+        bitCount += 2;
+        m_aomFilmGrain->cb_scaling_offset = 0;
+        bitCount += 8;
         for (int i = 0; i < m_aomFilmGrain->m_num_cb_points; i++)
         {
             for (int j = 0; j < 2; j++)
             {
                 x265_fread((char*)&m_aomFilmGrain->m_scaling_points_cbij, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+                bitCount+=8;
             }
         }
     }
     x265_fread((char*)&m_aomFilmGrain->m_num_cr_points, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=4;
     if (m_aomFilmGrain->m_num_cr_points)
     {
+        m_aomFilmGrain->point_cr_value_increment_bits = 8;
+        bitCount += 3;
+        m_aomFilmGrain->point_cr_scaling_bits = 8;
+        bitCount += 2;
+        m_aomFilmGrain->cr_scaling_offset = 0;
+        bitCount += 8;
         for (int i = 0; i < m_aomFilmGrain->m_num_cr_points; i++)
         {
             for (int j = 0; j < 2; j++)
             {
                 x265_fread((char*)&m_aomFilmGrain->m_scaling_points_crij, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+                bitCount+=8;
             }
         }
     }
     x265_fread((char*)&m_aomFilmGrain->m_scaling_shift, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=2;
     x265_fread((char*)&m_aomFilmGrain->m_ar_coeff_lag, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=2;
     if (m_aomFilmGrain->m_num_y_points)
     {
-
+        bitCount += 2;
         for (int i = 0; i < 24; i++)
         {
             x265_fread((char*)&m_aomFilmGrain->m_ar_coeffs_yi, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+            bitCount+=8;
         }
     }
     if (m_aomFilmGrain->m_num_cb_points || m_afg.m_chroma_scaling_from_luma)
     {
+        bitCount += 2;
         for (int i = 0; i < 25; i++)
         {
             x265_fread((char*)&m_aomFilmGrain->m_ar_coeffs_cbi, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+            bitCount+=8;
         }
     }
     if (m_aomFilmGrain->m_num_cr_points || m_afg.m_chroma_scaling_from_luma)
     {
-
+        bitCount += 2;
         for (int i = 0; i < 25; i++)
         {
             x265_fread((char*)&m_aomFilmGrain->m_ar_coeffs_cri, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+            bitCount+=8;
         }
     }
     x265_fread((char*)&m_aomFilmGrain->m_ar_coeff_shift, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=2;
     x265_fread((char*)&m_aomFilmGrain->m_grain_scale_shift, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount+=2;
     if (m_aomFilmGrain->m_num_cb_points)
     {
         x265_fread((char*)&m_aomFilmGrain->m_cb_mult, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 8;
         x265_fread((char*)&m_aomFilmGrain->m_cb_luma_mult, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 8;
         x265_fread((char*)&m_aomFilmGrain->m_cb_offset, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 9;
     }
     if (m_aomFilmGrain->m_num_cr_points)
     {
         x265_fread((char*)&m_aomFilmGrain->m_cr_mult, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 8;
         x265_fread((char*)&m_aomFilmGrain->m_cr_luma_mult, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 8;
         x265_fread((char*)&m_aomFilmGrain->m_cr_offset, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+        bitCount += 9;
     }
     x265_fread((char*)&m_aomFilmGrain->m_overlap_flag, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount++;
     x265_fread((char*)&m_aomFilmGrain->m_clip_to_restricted_range, sizeof(int32_t), 1, Aomfilmgrain, errorMessage);
+    bitCount++;
+
+    m_aomFilmGrain->luma_only_flag = m_aomFilmGrain->m_num_cb_points == 0 && m_aomFilmGrain->m_num_cr_points == 0;
+    bitCount++;
+    m_aomFilmGrain->subsamplingX = CHROMA_H_SHIFT(m_param->internalCsp);
+    m_aomFilmGrain->subsamplingY = CHROMA_V_SHIFT(m_param->internalCsp);
+    if (!m_aomFilmGrain->luma_only_flag)
+        bitCount += 2; // subsampling_x(1) + subsampling_y(1)
+    compute_film_grain_resolution(m_param->sourceWidth, m_param->sourceHeight, m_aomFilmGrain->units_resolution_log2,
+        m_aomFilmGrain->horz_resolution, m_aomFilmGrain->vert_resolution);
+    bitCount += 28; // apply_units_resolution_log2(4) + apply_horz_resolution(12) + apply_vert_resolution(12)
+    m_aomFilmGrain->predict_scaling_flag = 0;
+    bitCount++;
+    m_aomFilmGrain->predict_y_scaling_flag = 0;
+    m_aomFilmGrain->predict_cb_scaling_flag = 0;
+    m_aomFilmGrain->predict_cr_scaling_flag = 0;
+    m_aomFilmGrain->m_bitDepth = m_param->internalBitDepth;
+    bitCount++; // videosingnaltypepresentflag
+    if (m_frame0->m_encData->m_slice->m_sps->vuiParameters.videoSignalTypePresentFlag) bitCount += 4; // bit_depth_minus8(3) + cicp_info_present_flag(1)
+    if (m_frame0->m_encData->m_slice->m_sps->vuiParameters.colourDescriptionPresentFlag) bitCount += 25; // colourPrimaries(8) + transferCharacteristics(8) + matrixCoefficients(8)+ videoFullRangeFlag(1)
+    if (!m_aomFilmGrain->luma_only_flag) {
+        m_aomFilmGrain->m_chroma_scaling_from_luma = 0;
+        bitCount++;
+    }
+
+    m_aomFilmGrain->payload_size = (bitCount + 8 - 1) / 8;
+    m_aomFilmGrain->payload_bits = m_aomFilmGrain->payload_size < 4 ? 2 : 8;
+    bitCount += m_aomFilmGrain->payload_bits;
+    m_aomFilmGrain->payload_size = (bitCount + 8 - 1) / 8;
 }
 
 #if ENABLE_LIBVMAF
@@ -2435,7 +2583,7 @@
 
 Frame** FrameEncoder::getEncodedPicture(NALList& output)
 {
-    if (m_frame0 && (m_param->numLayers <= 1 || m_frame1))
+    if (m_frame0 && (m_param->numLayers <= 1 || (MAX_LAYERS > 1 && m_frame1)))
     {
         /* block here until worker thread completes */
         m_done.wait();

x265-4.1.tar/source/encoder/frameencoder.h -> x265-4.2.tar/source/encoder/frameencoder.h Changed

@@ -41,6 +41,8 @@
 #include "reference.h"
 #include "nal.h"
 #include "temporalfilter.h"
+#include "threadedme.h"
+#include <queue>
 
 namespace X265_NS {
 // private x265 namespace
@@ -241,6 +243,9 @@
     int64_t                  m_slicetypeWaitTimeMAX_LAYERS;        // total elapsed time waiting for decided frame
     int64_t                  m_totalWorkerElapsedTimeMAX_LAYERS;   // total elapsed time spent by worker threads processing CTUs
     int64_t                  m_totalNoWorkerTimeMAX_LAYERS;        // total elapsed time without any active worker threads
+    int64_t                  m_totalThreadedMEWaitMAX_LAYERS;      // total time spent waiting by CTUs for ThreadedME
+    int64_t                  m_totalThreadedMETimeMAX_LAYERS;      // total time spent processing by ThreadedME
+
 #if DETAILED_CU_STATS
     CUStats                  m_cuStats;
 #endif
@@ -267,6 +272,19 @@
 
     int                      m_sLayerId;
 
+    std::queue<CTUTask>      m_tmeTasks;
+    Lock                     m_tmeTasksLock;
+
+    struct TMEDependencyState
+    {
+        bool internal;
+        bool external;
+        bool isQueued;
+    };
+
+    std::vector<TMEDependencyState> m_tmeDeps;
+    Lock                     m_tmeDepLock;
+
     class WeightAnalysis : public BondedTaskGroup
     {
     public:

x265-4.1.tar/source/encoder/framefilter.cpp -> x265-4.2.tar/source/encoder/framefilter.cpp Changed

x265-4.1.tar/source/encoder/level.cpp -> x265-4.2.tar/source/encoder/level.cpp Changed

@@ -31,7 +31,7 @@
 typedef struct
 {
     uint32_t maxLumaSamples;
-    uint32_t maxLumaSamplesPerSecond;
+    uint64_t maxLumaSamplesPerSecond;
     uint32_t maxBitrateMain;
     uint32_t maxBitrateHigh;
     uint32_t maxCpbSizeMain;
@@ -44,31 +44,26 @@
 
 LevelSpec levels =
 {
-    { 36864,    552960,     128,      MAX_UINT, 350,    MAX_UINT, 2, Level::LEVEL1,   "1",   10 },
-    { 122880,   3686400,    1500,     MAX_UINT, 1500,   MAX_UINT, 2, Level::LEVEL2,   "2",   20 },
-    { 245760,   7372800,    3000,     MAX_UINT, 3000,   MAX_UINT, 2, Level::LEVEL2_1, "2.1", 21 },
-    { 552960,   16588800,   6000,     MAX_UINT, 6000,   MAX_UINT, 2, Level::LEVEL3,   "3",   30 },
-    { 983040,   33177600,   10000,    MAX_UINT, 10000,  MAX_UINT, 2, Level::LEVEL3_1, "3.1", 31 },
-    { 2228224,  66846720,   12000,    30000,    12000,  30000,    4, Level::LEVEL4,   "4",   40 },
-    { 2228224,  133693440,  20000,    50000,    20000,  50000,    4, Level::LEVEL4_1, "4.1", 41 },
-    { 8912896,  267386880,  25000,    100000,   25000,  100000,   6, Level::LEVEL5,   "5",   50 },
-    { 8912896,  534773760,  40000,    160000,   40000,  160000,   8, Level::LEVEL5_1, "5.1", 51 },
-    { 8912896,  1069547520, 60000,    240000,   60000,  240000,   8, Level::LEVEL5_2, "5.2", 52 },
-    { 35651584, 1069547520, 60000,    240000,   60000,  240000,   8, Level::LEVEL6,   "6",   60 },
-    { 35651584, 2139095040, 120000,   480000,   120000, 480000,   8, Level::LEVEL6_1, "6.1", 61 },
-    { 35651584, 4278190080U, 240000,  800000,   240000, 800000,   6, Level::LEVEL6_2, "6.2", 62 },
-    { MAX_UINT, MAX_UINT, MAX_UINT, MAX_UINT, MAX_UINT, MAX_UINT, 1, Level::LEVEL8_5, "8.5", 85 },
+    { 36864,      552960,         128,      MAX_UINT, 350,     MAX_UINT, 2, Level::LEVEL1,   "1",   10 },
+    { 122880,     3686400,        1500,     MAX_UINT, 1500,    MAX_UINT, 2, Level::LEVEL2,   "2",   20 },
+    { 245760,     7372800,        3000,     MAX_UINT, 3000,    MAX_UINT, 2, Level::LEVEL2_1, "2.1", 21 },
+    { 552960,     16588800,       6000,     MAX_UINT, 6000,    MAX_UINT, 2, Level::LEVEL3,   "3",   30 },
+    { 983040,     33177600,       10000,    MAX_UINT, 10000,   MAX_UINT, 2, Level::LEVEL3_1, "3.1", 31 },
+    { 2228224,    66846720,       12000,    30000,    12000,   30000,    4, Level::LEVEL4,   "4",   40 },
+    { 2228224,    133693440,      20000,    50000,    20000,   50000,    4, Level::LEVEL4_1, "4.1", 41 },
+    { 8912896,    267386880,      25000,    100000,   25000,   100000,   6, Level::LEVEL5,   "5",   50 },
+    { 8912896,    534773760,      40000,    160000,   40000,   160000,   8, Level::LEVEL5_1, "5.1", 51 },
+    { 8912896,    1069547520,     60000,    240000,   60000,   240000,   8, Level::LEVEL5_2, "5.2", 52 },
+    { 35651584,   1069547520,     60000,    240000,   60000,   240000,   8, Level::LEVEL6,   "6",   60 },
+    { 35651584,   2139095040,     120000,   480000,   120000,  480000,   8, Level::LEVEL6_1, "6.1", 61 },
+    { 35651584,   4278190080U,    240000,   800000,   240000,  800000,   6, Level::LEVEL6_2, "6.2", 62 },
+    { 80216064,   4812963840ULL,  320000,   1600000,  240000,  1600000,  6, Level::LEVEL6_3, "6.3", 63 },
+    { 142606336,  4812963840ULL,  320000,   1600000,  240000,  1600000,  6, Level::LEVEL7,   "7",   70 },
+    { 142606336,  8556380160ULL,  480000,   3200000,  480000,  3200000,  6, Level::LEVEL7_1, "7.1", 71 },
+    { 142606336,  17112760320ULL, 960000,   6400000,  960000,  6400000,  6, Level::LEVEL7_2, "7.2", 72 },
+    { MAX_UINT, MAX_UINT64, MAX_UINT, MAX_UINT, MAX_UINT, MAX_UINT, 1, Level::LEVEL8_5, "8.5", 85 },
 };
 
-static inline int _confirm(x265_param* param, bool bflag, const char* message)
-{
-    if (!bflag)
-        return 0;
-
-    x265_log(param, X265_LOG_ERROR, "%s\n", message);
-    return 1;
-}
-
 /* determine minimum decoder level required to decode the described video */
 void determineLevel(const x265_param &param, VPS& vps)
 {
@@ -153,7 +148,7 @@
 #endif
 
     uint32_t lumaSamples = param.sourceWidth * param.sourceHeight;
-    uint32_t samplesPerSec = (uint32_t)(lumaSamples * ((double)param.fpsNum / param.fpsDenom));
+    uint64_t samplesPerSec = (uint64_t)(lumaSamples * ((double)param.fpsNum / param.fpsDenom));
     uint32_t bitrate = param.rc.vbvMaxBitrate ? param.rc.vbvMaxBitrate : param.rc.bitrate;
 
     const uint32_t MaxDpbPicBuf = param.bEnableSCC ? 7 : 6;
@@ -164,9 +159,9 @@
     uint32_t i;
     if (param.bLossless)
     {
-        i = 13;
+        i = NumLevels - 1;
         vps.ptl.minCrForLevel = 1;
-        vps.ptl.maxLumaSrForLevel = MAX_UINT;
+        vps.ptl.maxLumaSrForLevel = MAX_UINT64;
         vps.ptl.levelIdc = Level::LEVEL8_5;
         vps.ptl.tierFlag = Level::MAIN;
     }
@@ -401,7 +396,7 @@
     bool allowHighTier = l.maxBitrateHigh < MAX_UINT && param.bHighTier;
 
     uint32_t lumaSamples = param.sourceWidth * param.sourceHeight;
-    uint32_t samplesPerSec = (uint32_t)(lumaSamples * ((double)param.fpsNum / param.fpsDenom));
+    uint64_t samplesPerSec = (uint64_t)(lumaSamples * ((double)param.fpsNum / param.fpsDenom));
     bool ok = true;
     if (lumaSamples > l.maxLumaSamples)
         ok = false;

x265-4.1.tar/source/encoder/motion.cpp -> x265-4.2.tar/source/encoder/motion.cpp Changed

@@ -628,6 +628,155 @@
     }
 }
 
+int MotionEstimate::diamondSearch(ReferencePlanes* ref, const MV& mvmin, const MV& mvmax, MV& outMV)
+{
+    int bcost = INT_MAX;
+    MV bmv(0, 0);
+    MV omv = bmv;
+
+    ALIGN_VAR_16(int, costs16);
+
+    intptr_t stride = ref->lumaStride;
+    pixel* fenc = fencPUYuv.m_buf0;
+    pixel* fref = ref->fpelPlane0 + blockOffset;
+
+    for (int16_t dist = 1; dist <= 4; dist <<= 1)
+    {
+        const int32_t top = omv.y - dist;
+        const int32_t bottom = omv.y + dist;
+        const int32_t left = omv.x - dist;
+        const int32_t right = omv.x + dist;
+        const int32_t top2 = omv.y - (dist >> 1);
+        const int32_t bottom2 = omv.y + (dist >> 1);
+        const int32_t left2 = omv.x - (dist >> 1);
+        const int32_t right2 = omv.x + (dist >> 1);
+
+        if (top >= mvmin.y && left >= mvmin.x && right <= mvmax.x && bottom <= mvmax.y)
+        {
+            COST_MV_X4(omv.x, top, omv.x, bottom, left, omv.y, right, omv.y);
+            COST_MV_X4(left2, top2, right2, top2, left2, bottom2, right2, bottom2);
+        }
+        else // check border for each mv
+        {
+            if (top >= mvmin.y) // check top
+            {
+                COST_MV(omv.x, top);
+            }
+            if (top2 >= mvmin.y) // check half top
+            {
+                if (left2 >= mvmin.x)  // check half left
+                {
+                    COST_MV(left2, top2);
+                }
+                if (right2 <= mvmax.x) // check half right
+                {
+                    COST_MV(right2, top2);
+                }
+            }
+            if (left >= mvmin.x) // check left
+            {
+                COST_MV(left, omv.y);
+            }
+            if (right <= mvmax.x) // check right
+            {
+                COST_MV(right, omv.y);
+            }
+            if (bottom2 <= mvmax.y) // check half bottom
+            {
+                if (left2 >= mvmin.x) // check half left
+                {
+                    COST_MV(left2, bottom2);
+                }
+                if (right2 <= mvmax.x) // check half right
+                {
+                    COST_MV(right2, bottom2);
+                }
+            }
+            if (bottom <= mvmax.y) // check bottom
+            {
+                COST_MV(omv.x, bottom);
+            }
+        }
+    }
+
+    for (int16_t dist = 8; dist <= 64; dist += 8)
+    {
+        const int32_t top = omv.y - dist;
+        const int32_t bottom = omv.y + dist;
+        const int32_t left = omv.x - dist;
+        const int32_t right = omv.x + dist;
+
+        if (top >= mvmin.y && left >= mvmin.x && right <= mvmax.x && bottom <= mvmax.y)
+        {
+            COST_MV_X4(omv.x, top, left, omv.y, right, omv.y, omv.x, bottom);
+
+            for (int16_t index = 1; index < 4; index++)
+            {
+                int32_t posYT = top + ((dist >> 2) * index);
+                int32_t posYB = bottom - ((dist >> 2) * index);
+                int32_t posXL = omv.x - ((dist >> 2) * index);
+                int32_t posXR = omv.x + ((dist >> 2) * index);
+
+                COST_MV_X4(posXL, posYT,
+                    posXR, posYT,
+                    posXL, posYB,
+                    posXR, posYB);
+            }
+        }
+        else // check border for each mv
+        {
+            if (top >= mvmin.y) // check top
+            {
+                COST_MV(omv.x, top);
+            }
+            if (left >= mvmin.x) // check left
+            {
+                COST_MV(left, omv.y);
+            }
+            if (right <= mvmax.x) // check right
+            {
+                COST_MV(right, omv.y);
+            }
+            if (bottom <= mvmax.y) // check bottom
+            {
+                COST_MV(omv.x, bottom);
+            }
+            for (int16_t index = 1; index < 4; index++)
+            {
+                int32_t posYT = top + ((dist >> 2) * index);
+                int32_t posYB = bottom - ((dist >> 2) * index);
+                int32_t posXL = omv.x - ((dist >> 2) * index);
+                int32_t posXR = omv.x + ((dist >> 2) * index);
+
+                if (posYT >= mvmin.y) // check top
+                {
+                    if (posXL >= mvmin.x) // check left
+                    {
+                        COST_MV(posXL, posYT);
+                    }
+                    if (posXR <= mvmax.x) // check right
+                    {
+                        COST_MV(posXR, posYT);
+                    }
+                }
+                if (posYB <= mvmax.y) // check bottom
+                {
+                    if (posXL >= mvmin.x) // check left
+                    {
+                        COST_MV(posXL, posYB);
+                    }
+                    if (posXR <= mvmax.x) // check right
+                    {
+                        COST_MV(posXR, posYB);
+                    }
+                }
+            }
+        }
+    }
+    outMV = bmv;
+    return bcost;
+}
+
 void MotionEstimate::refineMV(ReferencePlanes* ref,
                               const MV&        mvmin,
                               const MV&        mvmax,
@@ -1596,6 +1745,14 @@
     // check mv range for slice bound
     X265_CHECK(((bmv.y >= qmvmin.y) & (bmv.y <= qmvmax.y)), "mv beyond range!");
 
+    // Get a chance to ZeroMv
+    if (bmv.notZero())
+    {
+      int cost = subpelCompare(ref, MV(0, 0), satd) + mvcost(MV(0, 0));
+      if (cost <= bcost)
+        bmv = MV(0, 0);
+    }
+
     x265_emms();
     outQMv = bmv;
     return bcost;

x265-4.1.tar/source/encoder/motion.h -> x265-4.2.tar/source/encoder/motion.h Changed

x265-4.1.tar/source/encoder/ratecontrol.cpp -> x265-4.2.tar/source/encoder/ratecontrol.cpp Changed

@@ -71,7 +71,7 @@
     }\
 }
 
-inline int calcScale(uint32_t x)
+inline int calcScale(uint64_t x)
 {
     static uint8_t lut16 = {4, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0};
     int y, z = (((x & 0xffff) - 1) >> 27) & 16;
@@ -254,15 +254,16 @@
     m_relativeComplexity = NULL;
 
     // vbv initialization
-    m_param->rc.vbvBufferSize = x265_clip3(0, 2000000, m_param->rc.vbvBufferSize);
-    m_param->rc.vbvMaxBitrate = x265_clip3(0, 2000000, m_param->rc.vbvMaxBitrate);
-    m_param->rc.vbvBufferInit = x265_clip3(0.0, 2000000.0, m_param->rc.vbvBufferInit);
-    m_param->vbvBufferEnd = x265_clip3(0.0, 2000000.0, m_param->vbvBufferEnd);
+    m_param->rc.vbvBufferSize = x265_clip3(0, 8000000, m_param->rc.vbvBufferSize);
+    m_param->rc.vbvMaxBitrate = x265_clip3(0, 8000000, m_param->rc.vbvMaxBitrate);
+    m_param->rc.vbvBufferInit = x265_clip3(0.0, 8000000.0, m_param->rc.vbvBufferInit);
+    m_param->vbvBufferEnd = x265_clip3(0.0, 8000000.0, m_param->vbvBufferEnd);
     m_initVbv = false;
     m_singleFrameVbv = 0;
     m_rateTolerance = 1.0;
     m_encodedSegmentBits = 0;
     m_segDur = 0;
+    m_totalframesInSegment = 0;
 
     if (m_param->rc.vbvBufferSize)
     {
@@ -321,6 +322,7 @@
     m_leadingNoBSatd = 0;
     m_ipOffset = 6.0 * X265_LOG2(m_param->rc.ipFactor);
     m_pbOffset = 6.0 * X265_LOG2(m_param->rc.pbFactor);
+    m_iBits = 0;
 
     for (int i = 0; i < QP_MAX_MAX; i++)
         m_qpToEncodedBitsi = 0;
@@ -336,6 +338,8 @@
         m_lastQScaleFori = x265_qp2qScale(m_param->rc.rateControlMode == X265_RC_CRF ? CRF_INIT_QP : ABR_INIT_QP_MIN);
         m_lmini = x265_qp2qScale(m_param->rc.qpMin);
         m_lmaxi = x265_qp2qScale(m_param->rc.qpMax);
+        m_frameCountSegi = 0;
+        m_movingSumComplexitySegi = 0;
     }
 
     if (m_param->rc.rateControlMode == X265_RC_CQP)
@@ -407,8 +411,8 @@
         x265_log(m_param, X265_LOG_WARNING, "VBV buffer size cannot be smaller than one frame, using %d kbit\n",
             m_param->rc.vbvBufferSize);
     }
-    int vbvBufferSize = m_param->rc.vbvBufferSize * 1000;
-    int vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000;
+    uint64_t vbvBufferSize = m_param->rc.vbvBufferSize * 1000ULL;
+    uint64_t vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000ULL;
 
     if (m_param->bEmitHRDSEI && !m_param->decoderVbvMaxRate)
     {
@@ -416,9 +420,9 @@
         vbvBufferSize = hrd->cpbSizeValue << (hrd->cpbSizeScale + CPB_SHIFT);
         vbvMaxBitrate = hrd->bitRateValue << (hrd->bitRateScale + BR_SHIFT);
     }
-    m_bufferRate = vbvMaxBitrate / m_fps;
-    m_vbvMaxRate = vbvMaxBitrate;
-    m_bufferSize = vbvBufferSize;
+    m_bufferRate = static_cast<double>(vbvMaxBitrate) / m_fps;
+    m_vbvMaxRate = static_cast<double>(vbvMaxBitrate);
+    m_bufferSize = static_cast<double>(vbvBufferSize);
     m_singleFrameVbv = m_bufferRate * 1.1 > m_bufferSize;
 
     if (m_param->rc.vbvBufferInit > 1.)
@@ -492,7 +496,11 @@
     m_accumPNorm = .01;
     m_accumPQp = (m_param->rc.rateControlMode == X265_RC_CRF ? CRF_INIT_QP : ABR_INIT_QP_MIN) * m_accumPNorm;
 
-
+    for (int i = 0; i < 3; i++)
+    {
+        m_frameCountSegi = 0;
+        m_movingSumComplexitySegi = 0;
+    }
     /* Frame Predictors used in vbv */
     initFramePredictors();
     if (!m_statFileOut && (m_param->rc.bStatWrite || m_param->rc.bStatRead))
@@ -664,7 +672,6 @@
                 }
                 /* read stats */
                 p = statsIn;
-                double totalQpAq = 0;
                 for (int i = 0; i < m_numEntries; i++)
                 {
                     RateControlEntry *rce, *rcePocOrder;
@@ -729,7 +736,6 @@
                         return false;
                     }
                     rce->qScale = rce->newQScale = x265_qp2qScale(qpRc);
-                    totalQpAq += qpAq;
                     rce->qpNoVbv = qNoVbv;
                     rce->qpaRc = qpRc;
                     rce->qpAq = qpAq;
@@ -828,8 +834,8 @@
 {
     if (m_isVbv)
     {
-        m_param->rc.vbvBufferSize = x265_clip3(0, 2000000, m_param->rc.vbvBufferSize);
-        m_param->rc.vbvMaxBitrate = x265_clip3(0, 2000000, m_param->rc.vbvMaxBitrate);
+        m_param->rc.vbvBufferSize = x265_clip3(0, 8000000, m_param->rc.vbvBufferSize);
+        m_param->rc.vbvMaxBitrate = x265_clip3(0, 8000000, m_param->rc.vbvMaxBitrate);
         if (m_param->reconfigWindowSize)
             m_param->rc.vbvMaxBitrate = (int)(m_param->rc.vbvMaxBitrate * (double)(m_fps / m_param->reconfigWindowSize));
         if (m_param->rc.vbvMaxBitrate < m_param->rc.bitrate &&
@@ -845,11 +851,11 @@
             x265_log(m_param, X265_LOG_WARNING, "VBV buffer size cannot be smaller than one frame, using %d kbit\n",
                 m_param->rc.vbvBufferSize);
         }
-        int vbvBufferSize = m_param->rc.vbvBufferSize * 1000;
-        int vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000;
-        m_bufferRate = vbvMaxBitrate / m_fps;
-        m_vbvMaxRate = vbvMaxBitrate;
-        m_bufferSize = vbvBufferSize;
+        uint64_t vbvBufferSize = m_param->rc.vbvBufferSize * 1000ULL;
+        uint64_t vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000ULL;
+        m_bufferRate = static_cast<double>(vbvMaxBitrate) / m_fps;
+        m_vbvMaxRate = static_cast<double>(vbvMaxBitrate);
+        m_bufferSize = static_cast<double>(vbvBufferSize);
         m_singleFrameVbv = m_bufferRate * 1.1 > m_bufferSize;
     }
     if (m_param->rc.rateControlMode == X265_RC_CRF)
@@ -891,8 +897,8 @@
 
 void RateControl::initHRD(SPS& sps)
 {
-    int vbvBufferSize = m_param->rc.vbvBufferSize * 1000;
-    int vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000;
+    uint64_t vbvBufferSize = m_param->rc.vbvBufferSize * 1000ULL;
+    uint64_t vbvMaxBitrate = m_param->rc.vbvMaxBitrate * 1000ULL;
 
     // Init HRD
     HRDInfo* hrd = &sps.vuiParameters.hrdParameters;
@@ -904,12 +910,12 @@
 
     // normalize HRD size and rate to the value / scale notation
     hrd->bitRateScale = x265_clip3(0, 15, calcScale(vbvMaxBitrate) - BR_SHIFT);
-    hrd->bitRateValue = (vbvMaxBitrate >> (hrd->bitRateScale + BR_SHIFT));
+    hrd->bitRateValue = static_cast<uint32_t>(vbvMaxBitrate >> (hrd->bitRateScale + BR_SHIFT));
 
     hrd->cpbSizeScale = x265_clip3(0, 15, calcScale(vbvBufferSize) - CPB_SHIFT);
-    hrd->cpbSizeValue = (vbvBufferSize >> (hrd->cpbSizeScale + CPB_SHIFT));
-    int bitRateUnscale = hrd->bitRateValue << (hrd->bitRateScale + BR_SHIFT);
-    int cpbSizeUnscale = hrd->cpbSizeValue << (hrd->cpbSizeScale + CPB_SHIFT);
+    hrd->cpbSizeValue = static_cast<uint32_t>(vbvBufferSize >> (hrd->cpbSizeScale + CPB_SHIFT));
+    uint64_t bitRateUnscale = (uint64_t)hrd->bitRateValue << (hrd->bitRateScale + BR_SHIFT);
+    uint64_t cpbSizeUnscale = (uint64_t)hrd->cpbSizeValue << (hrd->cpbSizeScale + CPB_SHIFT);
 
     // arbitrary
     #define MAX_DURATION 0.5
@@ -1108,7 +1114,7 @@
 
 bool RateControl::initPass2()
 {
-    uint64_t allConstBits = 0, allCodedBits = 0;
+    uint64_t allConstBits = 0;
     uint64_t allAvailableBits = uint64_t(m_param->rc.bitrate * 1000. * m_numEntries * m_frameDuration);
     int startIndex, endIndex;
     int fps = X265_MIN(m_param->keyframeMax, (int)(m_fps + 0.5));
@@ -1127,7 +1133,6 @@
         for (endIndex = m_start; endIndex < m_numEntries; endIndex++)
         {
             allConstBits += m_rce2PassendIndex.miscBits;
-            allCodedBits += m_rce2PassendIndex.coeffBits + m_rce2PassendIndex.mvBits;
         }
 
         if (allAvailableBits < allConstBits)
@@ -1225,14 +1230,13 @@
     int t0, t1;
     double qScaleMin = x265_qp2qScale(m_param->rc.qpMin);
     double qScaleMax = x265_qp2qScale(m_param->rc.qpMax);
-    int iterations = 0 , adjMin, adjMax;
+    int adjMin, adjMax;
     CHECKED_MALLOC(fills, double, m_numEntries + 1);
     fills++;
 
     /* adjust overall stream size */
     do
     {
-        iterations++;
         prevBits = expectedBits;
 
         if (expectedBits)
@@ -1370,6 +1374,14 @@
             //Reset SBRC buffer
             m_encodedSegmentBits = 0;
             m_segDur = 0;
+            m_iBits = 0;
+            m_totalframesInSegment = m_param->totalFrames - m_framesDone;
+
+            for (int i = 0; i < 3; i++)
+            {
+                m_frameCountSegi = 0;
+                m_movingSumComplexitySegi = 0;
+            }
         }
     }
 
@@ -1863,14 +1875,10 @@
     }
 
     if (wantedBits > 0 && encodedBits > 0 && (!m_partialResidualFrames || 
-        m_param->rc.bStrictCbr || m_isGrainEnabled || (m_bRcReConfig && m_param->rc.rateControlMode == X265_RC_ABR)))
+        m_param->rc.bStrictCbr || m_isGrainEnabled))
     {
         abrBuffer *= X265_MAX(1, sqrt(timeDone));
         overflow = x265_clip3(.5, 2.0, 1.0 + (encodedBits - wantedBits) / abrBuffer);
-        if (m_bRcReConfig && overflow > 1.05)
-            qScale *= m_lstep;
-        if (m_bRcReConfig && overflow < 0.95)
-            qScale /= m_lstep;
         qScale *= overflow;
     }
     return qScale;
@@ -1911,7 +1919,7 @@
     int newQp = rcOverflow > 1.1 ? curQp + 2 : rcOverflow > 1 ? curQp + 1 : curQp - 1 ;
     double projectedBitrate =  int(m_fps + 0.5) * m_qpToEncodedBitsnewQp;
     if (curBitrate > 0 && projectedBitrate > 0)
-        q =  abs(projectedBitrate - m_bitrate) < abs (curBitrate - m_bitrate) ? x265_qp2qScale(newQp) : m_lastQScaleForP_SLICE;
+        q =  std::abs(projectedBitrate - m_bitrate) < std::abs (curBitrate - m_bitrate) ? x265_qp2qScale(newQp) : m_lastQScaleForP_SLICE;
     else
         q = rcOverflow > 1 ? qScaleAvg * qpstep : rcOverflow < 1 ?  qScaleAvg / qpstep : m_lastQScaleForP_SLICE;
     return q;
@@ -1920,7 +1928,6 @@
 double RateControl::rateEstimateQscale(Frame* curFrame, RateControlEntry *rce)
 {
     double q;
-
     if (m_2pass)
     {
         if (m_sliceType != rce->sliceType)
@@ -2097,13 +2104,6 @@
                     qScale = x265_clip3(lmin, lmax, qScale);
                 m_lastQScaleForm_sliceType = qScale;
             }
-
-            if (m_bRcReConfig && m_param->rc.rateControlMode == X265_RC_ABR)
-            {
-                qScale = tuneQscaleToUpdatedBitrate(curFrame, qScale);
-                rce->qpNoVbv = x265_qScale2qp(qScale);
-                m_lastQScaleForm_sliceType = qScale;
-            }
         }
 
         if (m_2pass)
@@ -2264,15 +2264,25 @@
                 if (m_param->bEnableSBRC)
                 {
                     double rfConstant = m_param->rc.rfConstant;
-                    if (m_currentSatd < rce->movingAvgSum)
+                    if (m_currentSatd < (0.7 *m_movingSumComplexitySegrce->sliceType))
                         rfConstant += 2;
+                    else if (m_currentSatd > (1.3 *m_movingSumComplexitySegrce->sliceType))
+                        rfConstant -= 1;
                     double ipOffset = (curFrame->m_lowres.bScenecut ? m_ipOffset : m_ipOffset / 2.0);
                     rfConstant = (rce->sliceType == I_SLICE ? rfConstant - ipOffset :
                         (rce->sliceType == B_SLICE ? rfConstant + m_pbOffset : rfConstant));
                     double mbtree_offset = m_param->rc.cuTree ? (1.0 - m_param->rc.qCompress) * 13.5 : 0;
-                    double qComp = (m_param->rc.cuTree && !m_param->rc.hevcAq) ? 0.99 : m_param->rc.qCompress;
-                    m_rateFactorConstant = pow(m_currentSatd, 1.0 - qComp) /
+                    double qComp = (m_param->rc.cuTree && !m_param->rc.hevcAq) ? 1.0 : m_param->rc.qCompress;
+                    double baseCplx = m_ncu * (m_param->bframes ? 120 : 80);
+                    m_rateFactorConstant = pow(baseCplx, 1.0 - qComp) /
                         x265_qp2qScale(rfConstant + mbtree_offset);
+
+                    if (IS_REFERENCED(curFrame))
+                    {
+                        double movingAvg = m_frameCountSegrce->sliceType ? ((m_movingSumComplexitySegrce->sliceType * m_frameCountSegrce->sliceType) + m_currentSatd) / (m_frameCountSegrce->sliceType + 1) : m_currentSatd;
+                        m_movingSumComplexitySegrce->sliceType = movingAvg;
+                        m_frameCountSegrce->sliceType++;
+                    }
                 }
                 q = getQScale(rce, m_rateFactorConstant);
                 x265_zone* zone = getZone();
@@ -2299,7 +2309,7 @@
                 }
                 double tunedQScale = tuneAbrQScaleFromFeedback(initialQScale);
                 overflow = tunedQScale / initialQScale;
-                q = (!m_partialResidualFrames || m_bRcReConfig) ? tunedQScale : initialQScale;
+                q = (!m_partialResidualFrames) ? tunedQScale : initialQScale;
                 bool isEncodeEnd = (m_param->totalFrames &&
                     m_framesDone > 0.75 * m_param->totalFrames) ? 1 : 0;
                 bool isEncodeBeg = m_framesDone < (int)(m_fps + 0.5);
@@ -2326,7 +2336,7 @@
                 {
                     lqmin = m_lastQScaleForm_sliceType / m_lstep;
                     lqmax = m_lastQScaleForm_sliceType * m_lstep;
-                    if (!m_partialResidualFrames || m_isGrainEnabled || m_bRcReConfig)
+                    if (!m_partialResidualFrames || m_isGrainEnabled)
                     {
                         if (overflow > 1.1 && m_framesDone > 3)
                             lqmax *= m_lstep;
@@ -2385,13 +2395,6 @@
                 rce->qpNoVbv = x265_qScale2qp(q);
             }
             q = clipQscale(curFrame, rce, q);
-
-            if (m_bRcReConfig && m_param->rc.rateControlMode == X265_RC_ABR)
-            {
-                q = tuneQscaleToUpdatedBitrate(curFrame, q);
-                rce->qpNoVbv = x265_qScale2qp(q);
-            }
-
             if (m_2pass)
                 rce->frameSizePlanned = qScale2bits(rce, q);
             else
@@ -2557,6 +2560,8 @@
     {
         double totalDuration = m_segDur;
         double frameBitsTotal = m_encodedSegmentBits + predictSize(&m_predm_predType, q, (double)m_currentSatd);
+        double lookaheadBits = 0;
+        double lookaheadDur = 0;
         for (int i = 0; i < depth; i++)
         {
             int type = curFrame->m_lowres.plannedTypei;
@@ -2567,51 +2572,37 @@
             int predType = getPredictorType(curFrame->m_lowres.plannedTypei, type);
             double curBits = predictSize(&m_predpredType, q, (double)satd);
             frameBitsTotal += curBits;
+            lookaheadBits += curBits;
+            lookaheadDur += m_frameDuration;
             totalDuration += m_frameDuration;
         }
         //Check for segment buffer overflow and adjust QP accordingly
         double segDur = m_param->keyframeMax / m_fps;
         double allowedSize = m_vbvMaxRate * segDur;
         double remDur = segDur - totalDuration;
-        double remainingBits = frameBitsTotal / totalDuration * remDur;
-        if (frameBitsTotal + remainingBits > 0.9 * allowedSize)
-            q = q * 1.01;
-        else
-            break;
-    }
-    return q;
-}
+        double remainingBits = frameBitsTotal;
+        int isLastSegmentInaEncode = m_totalframesInSegment < m_param->keyframeMax;
+        remainingBits = lookaheadBits / lookaheadDur * remDur;
 
-double RateControl::tuneQscaleToUpdatedBitrate(Frame* curFrame, double q)
-{
-    int depth = 18;
-    if (m_isVbv && m_currentSatd > 0 && curFrame)
-    {
-        for (int iterations = 0; iterations < 100; iterations++)
+		//Reduce over estimation due to I frame size
+        if (m_iBits == 0)
         {
-            int i;
-            double frameBitsTotal = m_fps * predictSize(&m_predm_predType, q, (double)m_currentSatd);
-            for (i = 0; i < depth; i++)
-            {
-                int type = curFrame->m_lowres.plannedTypei;
-                if (type == X265_TYPE_AUTO)
-                    break;
-                int64_t satd = curFrame->m_lowres.plannedSatdi >> (X265_DEPTH - 8);
-                type = IS_X265_TYPE_I(curFrame->m_lowres.plannedTypei) ? I_SLICE : IS_X265_TYPE_B(curFrame->m_lowres.plannedTypei) ? B_SLICE : P_SLICE;
-                int predType = getPredictorType(curFrame->m_lowres.plannedTypei, type);
-                double curBits = m_fps * predictSize(&m_predpredType, q, (double)satd);
-                frameBitsTotal += curBits;
-            }
-            frameBitsTotal /= i;
-            double allowedSize = (double)(curFrame->m_targetBitrate * 1000);
-            if (frameBitsTotal >= 1.1 * allowedSize)
-                q = q * 1.1;
-            else if (frameBitsTotal >= 1.05 * allowedSize)
-                q = q * 1.05;
-            else if (frameBitsTotal <= 0.9 * allowedSize)
-                q = q / 1.1;
-            else if (frameBitsTotal <= 0.95 * allowedSize)
-                q = q / 1.05;
+            //Compute number of I frames
+            double estimatedIBits = predictSize(&m_predm_predType, q, (double)m_currentSatd) * remDur;
+            remainingBits -= estimatedIBits;
+        }
+
+		if (!isLastSegmentInaEncode && m_segDur <  (0.2 * segDur))
+        {
+            if (frameBitsTotal + remainingBits > (allowedSize * 2))
+                q = q * 1.01;
+            else
+                break;
+        }
+        else
+        {
+            if (frameBitsTotal + remainingBits > (allowedSize))
+                q = q * 1.01;
             else
                 break;
         }
@@ -2656,8 +2647,7 @@
                     if (type == X265_TYPE_AUTO || totalDuration >= 1.0)
                         break;
                     totalDuration += m_frameDuration;
-                    double wantedFrameSize = ((m_bRcReConfig && m_param->rc.rateControlMode == X265_RC_ABR) ?
-                        curFrame->m_targetBitrate * 1000 : m_vbvMaxRate) * m_frameDuration;
+                    double wantedFrameSize = m_vbvMaxRate * m_frameDuration;
                     if (bufferFillCur + wantedFrameSize <= m_bufferSize)
                         bufferFillCur += wantedFrameSize;
                     int64_t satd = curFrame->m_lowres.plannedSatdj >> (X265_DEPTH - 8);
@@ -3291,6 +3281,10 @@
             rce->hrdTiming->dpbOutputTime = (double)rce->picTimingSEI->m_picDpbOutputDelay * time->numUnitsInTick / time->timeScale + rce->hrdTiming->cpbRemovalTime;
         }
     }
+    if (rce->sliceType == I_SLICE)
+    {
+        m_iBits = actualBits;
+    }
     rce->isActive = false;
     // Allow rateControlStart of next frame only when rateControlEnd of previous frame is over
     m_startEndOrder.incr();

x265-4.1.tar/source/encoder/ratecontrol.h -> x265-4.2.tar/source/encoder/ratecontrol.h Changed

@@ -178,7 +178,7 @@
     int     m_qpConstant3;
     int     m_lastNonBPictType;
     int     m_framesDone;        /* # of frames passed through RateCotrol already */
-
+    int64_t m_iBits;
     double  m_cplxrSum;          /* sum of bits*qscale/rceq */
     double  m_wantedBitsWindow;  /* target bitrate * window */
     double  m_accumPQp;          /* for determining I-frame quant */
@@ -194,6 +194,8 @@
     int64_t m_totalBits;        /* total bits used for already encoded frames (after ammortization) */
     int64_t m_encodedBits;      /* bits used for encoded frames (without ammortization) */
     int64_t m_encodedSegmentBits;      /* bits used for encoded frames in a segment*/
+    double  m_movingSumComplexitySeg3;
+    int     m_frameCountSeg3;
     double  m_segDur;
     double  m_fps;
     int64_t m_satdCostWindow50;
@@ -201,6 +203,7 @@
     int     m_sliderPos;
     int64_t m_lastRemovedSatdCost;
     double  m_movingAvgSum;
+    int     m_totalframesInSegment;
 
     /* To detect a pattern of low detailed static frames in single pass ABR using satdcosts */
     int64_t m_lastBsliceSatdCost;
@@ -302,8 +305,7 @@
     double rateEstimateQscale(Frame* pic, RateControlEntry *rce); // main logic for calculating QP based on ABR
     double tuneAbrQScaleFromFeedback(double qScale);
     double tuneQScaleForZone(RateControlEntry *rce, double qScale); // Tune qScale to adhere to zone budget
-    double tuneQscaleForSBRC(Frame* curFrame, double q); // Tune qScale to adhere to segment budget
-    double tuneQscaleToUpdatedBitrate(Frame* curFrame, double q); // Tune qScale according to updated bitrate
+    double tuneQscaleForSBRC(Frame* curFrame, double q); // Tune qScale to adhere to segment budget	
     void   accumPQpUpdate();
 
     int    getPredictorType(int lowresSliceType, int sliceType);

x265-4.1.tar/source/encoder/rdcost.h -> x265-4.2.tar/source/encoder/rdcost.h Changed

@@ -76,18 +76,13 @@
             qpCr = x265_clip3(QP_MIN, QP_MAX_SPEC, qp + slice.m_pps->chromaQpOffset1 + slice.m_chromaQpOffset1);
         }
 
-        if (slice.m_sps->chromaFormatIdc == X265_CSP_I444)
-        {
-            int chroma_offset_idx = X265_MIN(qp - qpCb + 12, MAX_CHROMA_LAMBDA_OFFSET);
-            uint16_t lambdaOffset = m_psyRd ? x265_chroma_lambda2_offset_tabchroma_offset_idx : 256;
-            m_chromaDistWeight0 = lambdaOffset;
+        int chroma_offset_idx = X265_MIN(qp - qpCb + 12, MAX_CHROMA_LAMBDA_OFFSET);
+        uint16_t lambdaOffset = x265_chroma_lambda2_offset_tabchroma_offset_idx;
+        m_chromaDistWeight0 = lambdaOffset;
 
-            chroma_offset_idx = X265_MIN(qp - qpCr + 12, MAX_CHROMA_LAMBDA_OFFSET);
-            lambdaOffset = m_psyRd ? x265_chroma_lambda2_offset_tabchroma_offset_idx : 256;
-            m_chromaDistWeight1 = lambdaOffset;
-        }
-        else
-            m_chromaDistWeight0 = m_chromaDistWeight1 = 256;
+        chroma_offset_idx = X265_MIN(qp - qpCr + 12, MAX_CHROMA_LAMBDA_OFFSET);
+        lambdaOffset = x265_chroma_lambda2_offset_tabchroma_offset_idx;
+        m_chromaDistWeight1 = lambdaOffset;
     }
 
     void setLambda(double lambda2, double lambda)

x265-4.1.tar/source/encoder/sao.cpp -> x265-4.2.tar/source/encoder/sao.cpp Changed

@@ -733,7 +733,6 @@
     const CUData* cu = m_frame->m_encData->getPicCTU(addr);
     const pixel* fenc0 = m_frame->m_fencPic->getPlaneAddr(plane, addr);
     const pixel* rec0  = reconPic->getPlaneAddr(plane, addr);
-    const pixel* fenc;
     const pixel* rec;
     intptr_t stride = plane ? reconPic->m_strideC : reconPic->m_stride;
     uint32_t picWidth  = m_param->sourceWidth;
@@ -863,7 +862,6 @@
                     skipR = 5;
                 }
 
-                fenc = fenc0;
                 rec  = rec0;
 
                 startX = !lpelx;
@@ -873,7 +871,6 @@
                 endY   = (bpely == picHeight) ? ctuHeight - 1 : ctuHeight - skipB + plane_offset;
                 if (startY)
                 {
-                    fenc += stride;
                     rec += stride;
                 }
 
@@ -888,7 +885,6 @@
                     skipB = 4;
                     skipR = 5;
                 }
-                fenc = fenc0;
                 rec  = rec0;
                 startX = !lpelx;
                 endX   = (rpelx == picWidth) ? ctuWidth - 1 : ctuWidth - skipR + plane_offset;
@@ -898,7 +894,6 @@
 
                 if (startY)
                 {
-                    fenc += stride;
                     rec += stride;
                 }

x265-4.1.tar/source/encoder/search.cpp -> x265-4.2.tar/source/encoder/search.cpp Changed

@@ -33,6 +33,7 @@
 
 #include "analysis.h"  // TLD
 #include "framedata.h"
+#include "encoder.h"
 
 using namespace X265_NS;
 
@@ -222,6 +223,340 @@
     return quantQP;
 }
 
+void Search::puMotionEstimation(const Slice* slice, const CUGeom& cuGeom, CUData& cu, PicYuv* fencPic, int puOffset, PartSize part, int areaIdx, int finalIdx, bool isMVP , const int* neighborIdx)
+{
+#ifdef DETAILED_CU_STATS
+    m_statscu.m_encData->m_frameEncoderID.countMotionEstimate++;
+#endif
+
+    int satdCost = 0;
+    int numPredDir = slice->isInterP() ? 1 : 2;
+    int searchRange = isMVP ? 32 : m_param->searchRange;
+
+    MV mvp(0,0);
+    MV mvzero(0,0);
+
+    MV mvc(MD_ABOVE_LEFT + 1) * 2 + 2;
+    MV amvpCand2MAX_NUM_REFAMVP_NUM_CANDS;
+
+    MotionData bestME2;
+    bestME0.cost = MAX_UINT;
+    bestME1.cost = MAX_UINT;
+
+    int numPart = cu.getNumPartInter(0);
+    uint32_t lastMode = 0;
+
+    int row = cu.m_cuAddr / m_slice->m_sps->numCuInWidth;
+    int col = cu.m_cuAddr % m_slice->m_sps->numCuInWidth;
+
+    int numMvc = 0;
+    for (int puIdx = 0; puIdx < numPart; puIdx++)
+    {
+        PredictionUnit pu(cu, cuGeom, puIdx);
+
+        int pos = finalIdx + puIdx * puOffset;
+        int slotIdx = (col % m_slice->m_sps->numCuInWidth) * m_slice->m_sps->numCuInHeight + row;
+
+        InterNeighbourMV neighbours6;
+        if(!isMVP)
+           cu.getNeighbourMV(puIdx, pu.puAbsPartIdx, neighbours);
+
+        for (int list = 0; list < numPredDir; list++)
+        {
+            int numIdx = slice->m_numRefIdxlist;
+            for (int ref = 0; ref < numIdx; ref++)
+            {
+                getBlkBits(part, slice->isInterP(), puIdx, lastMode, m_listSelBits);
+                uint32_t bits = m_listSelBitslist + MVP_IDX_BITS;
+                bits += getTUBits(ref, numIdx);
+
+                MV mvmin, mvmax, outmv,mvp_lowres;;
+                mvp = !isMVP ? m_areaBestMVareaIdxlistref : mvp;
+
+                MV zeroMV2 = {0,0};
+                const MV* amvp = zeroMV;
+                int mvpIdx = 0;
+
+                bool bLowresMVP = false;
+                if (!isMVP)
+                {
+                    for(int dir = MD_LEFT; dir <= MD_ABOVE_LEFT ; dir++)
+                    {
+                        int neighIdx = neighborIdxdir;
+                        if (neighIdx >= 0)
+                        {
+                            MEData& neighborData = slice->m_ctuMVslotIdx * MAX_NUM_PUS_PER_CTU + neighIdx;
+                            for (int i = 0; i < 2; i++)
+                            {
+                                neighboursdir.mvi = neighborData.mvi;
+                                neighboursdir.refIdxi = neighborData.refi;
+                            }
+                            neighboursdir.isAvailable = (neighborData.ref0 >= 0 || neighborData.ref1 >= 0);
+                        }
+                        else
+                        {
+                            for (int i = 0; i < 2; i++)
+                                neighboursdir.refIdxi = -1;
+                            neighboursdir.isAvailable = false;
+                        }
+                    }
+
+                    numMvc = cu.getPMV(neighbours, list, ref, amvpCandlistref, mvc);
+                    if (numMvc > 0)
+                    {
+                        amvp = amvpCandlistref;
+                        mvpIdx = selectMVP(cu, pu, amvp, list, ref);
+                        mvp = amvpmvpIdx;                 
+                    }
+                    else if (slice->m_refFrameListlistref->m_encData->m_slice->m_sliceType != I_SLICE)
+                    {
+                        MEData meData = slice->m_refFrameListlistref->m_encData->m_slice->m_ctuMVslotIdx * MAX_NUM_PUS_PER_CTU + pos;
+
+                        bool bi = (meData.ref0 >= 0 && meData.ref1 >= 0);
+                        bool uniL0 = (meData.ref0 >= 0 && meData.ref1 == REF_NOT_VALID);
+                        bool uniL1 = (meData.ref1 >= 0 && meData.ref0 == REF_NOT_VALID);
+
+                        if (uniL0)
+                            mvp = meData.mv0;
+                        else if (uniL1)
+                            mvp = meData.mv1;
+                        else if (bi)
+                            mvp = meData.mvlist;
+                    }
+                }
+
+                m_me.setMVP(mvp);
+
+                if (!strlen(m_param->analysisSave) && !strlen(m_param->analysisLoad))
+                {
+                    uint32_t blockX = cu.m_cuPelX + g_zscanToPelXpu.puAbsPartIdx + (pu.width  >> 1);
+                    uint32_t blockY = cu.m_cuPelY + g_zscanToPelYpu.puAbsPartIdx + (pu.height >> 1);
+
+                    if (blockX < m_slice->m_sps->picWidthInLumaSamples && blockY < m_slice->m_sps->picHeightInLumaSamples)
+                    {
+                        MV lmv = getLowresMV(cu, pu, list, ref);
+                        int layer = m_param->numViews > 1 ? m_frame->m_viewId : (m_param->numScalableLayers > 1) ? m_frame->m_sLayerId : 0;
+                        if (lmv.notZero() && !layer)
+                        {
+                            mvcnumMvc++ = lmv;
+                            bLowresMVP = true;
+                        }
+                        mvp_lowres = lmv;
+                    }
+                }
+
+                PicYuv* recon = slice->m_mreflistref.reconPic;
+                int offset = recon->getLumaAddr(cu.m_cuAddr, pu.cuAbsPartIdx + pu.puAbsPartIdx) - recon->getLumaAddr(0);
+
+                if (m_param->searchMethod == X265_SEA)
+                {
+                    for (int planes = 0; planes < INTEGRAL_PLANE_NUM; planes++)
+                        m_me.integralplanes = slice->m_refFrameListlistref->m_encData->m_meIntegralplanes + offset;
+                }
+
+                m_me.setSourcePU(fencPic->m_picOrg0, fencPic->m_stride, offset, pu.width, pu.height, m_param->searchMethod, m_param->subpelRefine);
+                setSearchRange(cu, mvp, searchRange, mvmin, mvmax);
+
+                if (isMVP)
+                {
+                    satdCost = m_me.diamondSearch(&slice->m_mreflistref, mvmin, mvmax, outmv);
+                    m_areaBestMVareaIdxlistref = outmv;
+                }
+                else
+                {
+                    m_vertRestriction = slice->m_refPOCListlistref == slice->m_poc;
+                    satdCost = m_me.motionEstimate(&slice->m_mreflistref, mvmin, mvmax, mvp, numMvc, mvc, m_param->searchRange, outmv, m_param->maxSlices, m_vertRestriction,
+                        m_param->bSourceReferenceEstimation ? m_slice->m_refFrameListlistref->m_fencPic->getLumaAddr(0) : 0);
+
+                    if (bLowresMVP && mvp_lowres.notZero() && mvp_lowres != mvp)
+                    {
+                        MV outmv_lowres;
+                        bLowresMVP = false;
+                        setSearchRange(cu, mvp_lowres, m_param->searchRange, mvmin, mvmax);
+                        int lowresMvCost = m_me.motionEstimate(&slice->m_mreflistref,  mvmin, mvmax, mvp_lowres, numMvc, mvc, m_param->searchRange,outmv_lowres, m_param->maxSlices,
+                            m_vertRestriction, m_param->bSourceReferenceEstimation ? m_slice->m_refFrameListlistref->m_fencPic->getLumaAddr(0): 0);
+
+                        if (lowresMvCost < satdCost)
+                        {
+                            outmv = outmv_lowres;
+                            satdCost = lowresMvCost;
+                            bLowresMVP = true;
+                        }
+                    }
+                }
+
+                bits += m_me.bitcost(outmv);
+                uint32_t mvCost = m_me.mvcost(outmv);
+                uint32_t cost = (satdCost - mvCost) + m_rdCost.getCost(bits);
+
+                if(!isMVP)
+                {
+                    if (bLowresMVP)
+                        updateMVP(mvp, outmv, bits, cost, mvp_lowres);
+
+                    mvp = checkBestMVP(amvp, outmv, mvpIdx, bits, cost);
+                }
+                if (cost < bestMElist.cost)
+                {
+                    bestMElist.mv = outmv;
+                    bestMElist.mvp = mvp;
+                    bestMElist.mvpIdx = 0;
+                    bestMElist.cost = cost;
+                    bestMElist.bits = bits;
+                    bestMElist.mvCost = mvCost;
+                    bestMElist.ref = ref;
+                }
+            }
+        }
+
+        if (isMVP)
+            return;
+
+        //Bi-Direction
+        MotionData bidir2;
+        uint32_t bidirCost = MAX_UINT;
+        int bidirBits = 0;
+        Yuv& tmpPredYuv = m_rqtcuGeom.depth.tmpPredYuv;
+
+        if (slice->isInterB() && !cu.isBipredRestriction() &&
+            cu.m_partSizepu.puAbsPartIdx != SIZE_2Nx2N && bestME0.cost != MAX_UINT && bestME1.cost != MAX_UINT && !isMVP)
+        {
+            bidir0 = bestME0;
+            bidir1 = bestME1;
+
+            if (m_me.bChromaSATD)
+            {
+                cu.m_mv0pu.puAbsPartIdx = bidir0.mv;
+                cu.m_refIdx0pu.puAbsPartIdx = (int8_t)bidir0.ref;
+                cu.m_mv1pu.puAbsPartIdx = bidir1.mv;
+                cu.m_refIdx1pu.puAbsPartIdx = (int8_t)bidir1.ref;
+                motionCompensation(cu, pu, tmpPredYuv, true, true);
+
+                satdCost = m_me.bufSATD(tmpPredYuv.getLumaAddr(pu.puAbsPartIdx), tmpPredYuv.m_size) +
+                    m_me.bufChromaSATD(tmpPredYuv, pu.puAbsPartIdx);
+            }
+            else
+            {
+                PicYuv* refPic0 = slice->m_refReconPicList0bestME0.ref;
+                PicYuv* refPic1 = slice->m_refReconPicList1bestME1.ref;
+                Yuv* bidirYuv = m_rqtcuGeom.depth.bidirPredYuv;
+
+                predInterLumaPixel(pu, bidirYuv0, *refPic0, bestME0.mv);
+                predInterLumaPixel(pu, bidirYuv1, *refPic1, bestME1.mv);
+                primitives.pum_me.partEnum.pixelavg_pp(tmpPredYuv.m_size % 64 == 0) && (bidirYuv0.m_size % 64 == 0) && (bidirYuv1.m_size % 64 == 0)(tmpPredYuv.m_buf0, tmpPredYuv.m_size, bidirYuv0.getLumaAddr(pu.puAbsPartIdx), bidirYuv0.m_size,
+                    bidirYuv1.getLumaAddr(pu.puAbsPartIdx), bidirYuv1.m_size, 32);
+                satdCost = m_me.bufSATD(tmpPredYuv.m_buf0, tmpPredYuv.m_size);
+            }
+
+            bidirBits = bestME0.bits + bestME1.bits + m_listSelBits2 - (m_listSelBits0 + m_listSelBits1);
+            bidirCost = satdCost + m_rdCost.getCost(bidirBits);
+
+            bool bTryZero = bestME0.mv.notZero() || bestME1.mv.notZero();
+            if (bTryZero)
+            {
+                MV mvmin, mvmax;
+                int merange = X265_MAX(m_param->sourceWidth, m_param->sourceHeight);
+                setSearchRange(cu, mvzero, merange, mvmin, mvmax);
+                mvmax.y += 2;
+                mvmin <<= 2;
+                mvmax <<= 2;
+
+                bTryZero &= bestME0.mvp.checkRange(mvmin, mvmax);
+                bTryZero &= bestME1.mvp.checkRange(mvmin, mvmax);
+            }
+            if (bTryZero)
+            {
+                if (m_me.bChromaSATD)
+                {
+                    cu.m_mv0pu.puAbsPartIdx = mvzero;
+                    cu.m_refIdx0pu.puAbsPartIdx = (int8_t)bidir0.ref;
+                    cu.m_mv1pu.puAbsPartIdx = mvzero;
+                    cu.m_refIdx1pu.puAbsPartIdx = (int8_t)bidir1.ref;
+                    motionCompensation(cu, pu, tmpPredYuv, true, true);
+
+                    satdCost = m_me.bufSATD(tmpPredYuv.getLumaAddr(pu.puAbsPartIdx), tmpPredYuv.m_size) +
+                        m_me.bufChromaSATD(tmpPredYuv, pu.puAbsPartIdx);
+                }
+                else
+                {
+                    const pixel* ref0 = m_slice->m_mref0bestME0.ref.getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx + pu.puAbsPartIdx);
+                    const pixel* ref1 = m_slice->m_mref1bestME1.ref.getLumaAddr(pu.ctuAddr, pu.cuAbsPartIdx + pu.puAbsPartIdx);
+                    intptr_t refStride = slice->m_mref00.lumaStride;
+                    primitives.pum_me.partEnum.pixelavg_pp(tmpPredYuv.m_size % 64 == 0) && (refStride % 64 == 0)(tmpPredYuv.m_buf0, tmpPredYuv.m_size, ref0, refStride, ref1, refStride, 32);
+                    satdCost = m_me.bufSATD(tmpPredYuv.m_buf0, tmpPredYuv.m_size);
+                }
+
+                MV mvp0 = bestME0.mvp;
+                int mvpIdx0 = bestME0.mvpIdx;
+                uint32_t bits0 = bestME0.bits - m_me.bitcost(bestME0.mv, mvp0) + m_me.bitcost(mvzero, mvp0);
+
+                MV mvp1 = bestME1.mvp;
+                int mvpIdx1 = bestME1.mvpIdx;
+                uint32_t bits1 = bestME1.bits - m_me.bitcost(bestME1.mv, mvp1) + m_me.bitcost(mvzero, mvp1);
+
+                uint32_t cost = satdCost + m_rdCost.getCost(bits0) + m_rdCost.getCost(bits1);
+
+                if (cost < bidirCost)
+                {
+                    bidir0.mv = mvzero;
+                    bidir1.mv = mvzero;
+                    bidir0.mvp = mvp0;
+                    bidir1.mvp = mvp1;
+                    bidir0.mvpIdx = mvpIdx0;
+                    bidir1.mvpIdx = mvpIdx1;
+                    bidirCost = cost;
+                    bidirBits = bits0 + bits1 + m_listSelBits2 - (m_listSelBits0 + m_listSelBits1);
+                }
+            }
+        }
+        MEData& outME = slice->m_ctuMVslotIdx * MAX_NUM_PUS_PER_CTU + pos;
+
+        outME.ref0 = REF_NOT_VALID;
+        outME.ref1 = REF_NOT_VALID;
+
+        if (bidirCost < bestME0.cost && bidirCost < bestME1.cost)
+        {
+            lastMode = 2;
+
+            outME.mv0 = bidir0.mv;
+            outME.mv1 = bidir1.mv;
+            outME.mvp0 = bidir0.mvp;
+            outME.mvp1 = bidir1.mvp;
+            outME.mvCost0 = bestME0.mvCost;
+            outME.mvCost1 = bestME1.mvCost;
+            outME.ref0 = bestME0.ref;
+            outME.ref1 = bestME1.ref;
+
+            outME.bits = bidirBits;
+            outME.cost = bidirCost;
+        }
+        else if (bestME0.cost <= bestME1.cost)
+        {
+            lastMode = 0;
+
+            outME.mv0 = bestME0.mv;
+            outME.mvp0 = bestME0.mvp;
+            outME.mvCost0 = bestME0.mvCost;
+            outME.cost = bestME0.cost;
+            outME.bits = bestME0.bits;
+            outME.ref0 = bestME0.ref;
+            outME.ref1 = REF_NOT_VALID;
+        }
+        else
+        {
+            lastMode = 1;
+
+            outME.mv1 = bestME1.mv;
+            outME.mvp1 = bestME1.mvp;
+            outME.mvCost1 = bestME1.mvCost;
+            outME.cost = bestME1.cost;
+            outME.bits = bestME1.bits;
+            outME.ref1 = bestME1.ref;
+            outME.ref0 = REF_NOT_VALID;
+        }
+    }
+}
+
 #if CHECKED_BUILD || _DEBUG
 void Search::invalidateContexts(int fromDepth)
 {
@@ -276,7 +611,6 @@
         return;
     }
 
-    uint32_t tuDepthC = tuDepth;
     uint32_t log2TrSizeC = log2TrSize - m_hChromaShift;
 
     if (log2TrSizeC < 2)
@@ -285,7 +619,6 @@
         if (absPartIdx & 3)
             return;
         log2TrSizeC = 2;
-        tuDepthC--;
     }
 
     uint32_t qtLayer = log2TrSize - 2;
@@ -2440,7 +2773,7 @@
             /* if no peer threads were bonded, fall back to doing unidirectional
              * searches ourselves without overhead of singleMotionEstimation() */
         }
-        if (bDoUnidir)
+        if (bDoUnidir && !m_param->bThreadedME)
         {
             interMode.bestMEpuIdx0.ref = interMode.bestMEpuIdx1.ref = -1;
             uint32_t refMask = refMaskspuIdx ? refMaskspuIdx : (uint32_t)-1;
@@ -2552,7 +2885,7 @@
 
         if (slice->isInterB() && !cu.isBipredRestriction() &&  /* biprediction is possible for this PU */
             cu.m_partSizepu.puAbsPartIdx != SIZE_2Nx2N &&    /* 2Nx2N biprediction is handled elsewhere */
-            bestME0.cost != MAX_UINT && bestME1.cost != MAX_UINT)
+            bestME0.cost != MAX_UINT && bestME1.cost != MAX_UINT && !m_param->bThreadedME)
         {
             bidir0 = bestME0;
             bidir1 = bestME1;
@@ -2652,8 +2985,107 @@
             }
         }
 
+        uint32_t bestCost = MAX_INT;
+        bool isMerge = false;
+        bool isBidir = false;
+        bool uniL0 = false;
+        bool uniL1 = false;
+
+        if (m_param->bThreadedME)
+        {
+            int cuSize = 1 << cu.m_log2CUSize0;
+
+            int lookupWidth = pu.width;
+            int lookupHeight = pu.height;
+
+            bool isAmp = cu.m_partSize0 >= SIZE_2NxnU;
+
+            if (isAmp)
+            {
+                if (cu.m_partSize0 == SIZE_2NxnU || cu.m_partSize0 == SIZE_2NxnD)
+                    lookupHeight = (puIdx) ? (pu.width - pu.height) : pu.height;
+                else
+                    lookupWidth = (puIdx) ? (pu.height - pu.width) : pu.width;
+            }
+
+            int startIdx = g_puStartIdxlookupWidth + lookupHeightstatic_cast<int>(cu.m_partSize0);
+
+            int alignWidth = isAmp ? cuSize : pu.width;
+            int alignHeight = isAmp ? cuSize : pu.height;
+
+            int numPUX = m_param->maxCUSize / alignWidth;
+            int numPUY = m_param->maxCUSize / alignHeight;
+
+            int puOffset = isAmp ? (puIdx * numPUX * numPUY) : (cu.m_partSize0 == SIZE_2NxN ? (puIdx * numPUX) : puIdx);
+ 
+            int relX = (cu.m_cuPelX / alignWidth) % numPUX;
+            int relY = (cu.m_cuPelY / alignHeight) % numPUY;
+
+            int index = startIdx + (relY * numPUX + relX) + puOffset;
+
+            int row = cu.m_cuAddr / m_slice->m_sps->numCuInWidth;
+            int col = cu.m_cuAddr % m_slice->m_sps->numCuInWidth;
+
+            int slotIdx = (col % m_slice->m_sps->numCuInWidth) * m_slice->m_sps->numCuInHeight + row;
+
+            MEData meData = slice->m_ctuMVslotIdx * MAX_NUM_PUS_PER_CTU + index;
+
+            bestME0.ref = meData.ref0;
+            bestME1.ref = meData.ref1;
+
+            isBidir = (bestME0.ref >= 0 && bestME1.ref >= 0);
+            uniL0 = (bestME0.ref >= 0 && bestME1.ref == REF_NOT_VALID);
+            uniL1 = (bestME1.ref >= 0 && bestME0.ref == REF_NOT_VALID);
+
+            if(isBidir)
+            {
+                cu.getPMV(interMode.interNeighbours, 0, bestME0.ref, interMode.amvpCand0bestME0.ref, mvc);
+                cu.getPMV(interMode.interNeighbours, 1, bestME1.ref, interMode.amvpCand1bestME1.ref, mvc);
+
+                bidir0.mv = meData.mv0;
+                bidir1.mv = meData.mv1;
+                bidir0.mvp = interMode.amvpCand0bestME0.ref0;
+                bidir1.mvp = interMode.amvpCand1bestME1.ref0;
+                bidir0.mvCost = meData.mvCost0;
+                bidir1.mvCost = meData.mvCost1;
+                bidirCost = meData.cost;
+                bidirBits = meData.bits;
+
+                bestCost = bidirCost;
+            }
+            else if (uniL0)
+            {
+                cu.getPMV(interMode.interNeighbours, 0, bestME0.ref, interMode.amvpCand0bestME0.ref, mvc);
+
+                bestME0.mv = meData.mv0;
+                bestME0.mvp = interMode.amvpCand0bestME0.ref0;
+                bestME0.mvCost = meData.mvCost0;
+                bestME0.cost = meData.cost;
+                bestME0.bits = meData.bits;
+
+                bestCost = bestME0.cost;
+            }
+            else if (uniL1)
+            {
+                cu.getPMV(interMode.interNeighbours, 1, bestME1.ref, interMode.amvpCand1bestME1.ref, mvc);
+
+                bestME1.mv = meData.mv1;
+                bestME1.mvp = interMode.amvpCand1bestME1.ref0;
+                bestME1.mvCost = meData.mvCost1;
+                bestME1.cost = meData.cost;
+                bestME1.bits = meData.bits;
+
+                bestCost = bestME1.cost;
+            }
+            else
+                x265_log(NULL, X265_LOG_ERROR, "Invalid ME mode");
+
+            if (mrgCost < bestCost)
+                isMerge = true;
+        }
+
         /* select best option and store into CU */
-        if (mrgCost < bidirCost && mrgCost < bestME0.cost && mrgCost < bestME1.cost)
+        if ((mrgCost < bidirCost && mrgCost < bestME0.cost && mrgCost < bestME1.cost) || isMerge)
         {
             cu.m_mergeFlagpu.puAbsPartIdx = true;
             cu.m_mvpIdx0pu.puAbsPartIdx = merge.index; /* merge candidate ID is stored in L0 MVP idx */
@@ -2665,7 +3097,7 @@
 
             totalmebits += merge.bits;
         }
-        else if (bidirCost < bestME0.cost && bidirCost < bestME1.cost)
+        else if ((bidirCost < bestME0.cost && bidirCost < bestME1.cost) || isBidir)
         {
             lastMode = 2;
 
@@ -2683,7 +3115,7 @@
 
             totalmebits += bidirBits;
         }
-        else if (bestME0.cost <= bestME1.cost)
+        else if ((bestME0.cost <= bestME1.cost) || uniL0)
         {
             lastMode = 0;
 
@@ -5717,12 +6149,10 @@
 
     uint32_t log2TrSizeC = log2TrSize - m_hChromaShift;
     uint32_t codeChroma = (m_csp != X265_CSP_I400 && m_frame->m_fencPic->m_picCsp != X265_CSP_I400) ? 1 : 0;
-    uint32_t tuDepthC = tuDepth;
     if (log2TrSizeC < 2)
     {
         X265_CHECK(log2TrSize == 2 && m_csp != X265_CSP_I444 && tuDepth, "invalid tuDepth\n");
         log2TrSizeC = 2;
-        tuDepthC--;
         codeChroma &= !(absPartIdx & 3);
     }

x265-4.1.tar/source/encoder/search.h -> x265-4.2.tar/source/encoder/search.h Changed

@@ -179,6 +179,7 @@
     int64_t  pmodeBlockTime;                    // elapsed worker time blocked for pmode batch completion
     int64_t  weightAnalyzeTime;                 // elapsed worker time analyzing reference weights
     int64_t  totalCTUTime;                      // elapsed worker time in compressCTU (includes pmode master)
+    int64_t  tmeTime;                           // elapsed worker time in threadedME
 
     uint32_t skippedMotionReferencesNUM_CU_DEPTH;
     uint32_t totalMotionReferencesNUM_CU_DEPTH;
@@ -195,6 +196,8 @@
     uint64_t countPModeTasks;
     uint64_t countPModeMasters;
     uint64_t countWeightAnalyze;
+    uint64_t countTmeTasks;
+    uint64_t countTmeBlockedCTUs;
     uint64_t totalCTUs;
 
     CUStats() { clear(); }
@@ -227,6 +230,7 @@
         pmodeBlockTime += other.pmodeBlockTime;
         weightAnalyzeTime += other.weightAnalyzeTime;
         totalCTUTime += other.totalCTUTime;
+        tmeTime += other.tmeTime;
 
         countIntraAnalysis += other.countIntraAnalysis;
         countMotionEstimate += other.countMotionEstimate;
@@ -236,6 +240,8 @@
         countPModeTasks += other.countPModeTasks;
         countPModeMasters += other.countPModeMasters;
         countWeightAnalyze += other.countWeightAnalyze;
+        countTmeTasks += other.countTmeTasks;
+        countTmeBlockedCTUs += other.countTmeBlockedCTUs;
         totalCTUs += other.totalCTUs;
 
         other.clear();
@@ -288,6 +294,8 @@
 
     bool            m_vertRestriction;
 
+    MV              m_areaBestMV52MAX_NUM_REF;
+
 #if ENABLE_SCC_EXT
     int             m_ibcEnabled;
     int             m_numBVs;
@@ -341,6 +349,16 @@
 
     MV getLowresMV(const CUData& cu, const PredictionUnit& pu, int list, int ref);
 
+    /**
+     * @brief Run motion estimation for one PU partition shape and persist the best ME result.
+     *
+     * Used by Analysis threaded-ME flow. With isMVP=true this bootstraps area MVPs,
+     * and with isMVP=false it performs full PU ME using spatial/temporal neighbors
+     * and stores results into per-CTU MV slots addressed by finalIdx/puOffset.
+     */
+    void puMotionEstimation(const Slice* slice, const CUGeom& cuGeom, CUData& ctu, PicYuv* fencPic, int puOffset, PartSize part, int areaIdx, int finalIdx, 
+        bool isMVP ,  const int* neighborIdx = NULL);
+ 
 #if ENABLE_SCC_EXT
     void      predInterSearch(Mode& interMode, const CUGeom& cuGeom, bool bChromaMC, uint32_t masks2, MV* iMVCandList = NULL);
     bool      predIntraBCSearch(Mode& intraBCMode, const CUGeom& cuGeom, bool bChromaMC, PartSize ePartSize, bool testOnlyPred, bool bUse1DSearchFor8x8, IBC& ibc);

x265-4.1.tar/source/encoder/sei.h -> x265-4.2.tar/source/encoder/sei.h Changed

@@ -204,83 +204,171 @@
     int32_t     m_chroma_scaling_from_luma;
     int32_t     m_grain_scale_shift;
     uint16_t    m_grain_seed;
+    bool        subsamplingX;
+    bool        subsamplingY;
+    bool        predict_scaling_flag;
+    bool        predict_y_scaling_flag;
+    bool        predict_cb_scaling_flag;
+    bool        predict_cr_scaling_flag;
+    int         units_resolution_log2;
+    int         horz_resolution;
+    int         vert_resolution;
+    bool        luma_only_flag;
+    int         point_y_value_increment_bits;
+    int         point_y_scaling_bits;
+    int         point_cb_value_increment_bits;
+    int         point_cb_scaling_bits;
+    int         point_cr_value_increment_bits;
+    int         point_cr_scaling_bits;
+    int         cb_scaling_offset;
+    int         cr_scaling_offset;
+    int         payload_size;
+    int         payload_bits;
 
-    void writeSEI(const SPS&)
+    void writeSEI(const SPS& sps)
     {
-        WRITE_CODE(0x26, 8, "country_code");
+        WRITE_CODE(0xB5, 8, "country_code");
         WRITE_CODE(0x5890, 16, "provider_code");
-        WRITE_CODE(0x0001, 16, "provider_oriented_code");
-        WRITE_FLAG(m_apply_grain, "afgs1_enable_flag");
-        WRITE_CODE(m_grain_seed, 16, "grain_seed");
+        WRITE_CODE(0x0001, 8, "provider_oriented_code");
+        WRITE_FLAG(1, "afgs1_enable_flag");
+        WRITE_CODE(0, 4, "reserved_4bits");
+        WRITE_CODE(0, 3, "num_film_grain_sets_minus1");
+        WRITE_FLAG(payload_size < 4 ? 1 : 0, "payload_less_than_4byte_flag");
+        WRITE_CODE(payload_size, payload_bits, "payload_size");
+
         WRITE_CODE(0, 3, "film_grain_param_set_idx");
-        WRITE_CODE(m_update_grain, 1, "update_grain");
-        WRITE_CODE(m_num_y_points, 4, "num_y_points");
-        if (m_num_y_points)
+        WRITE_FLAG(m_apply_grain, "apply_grain_flag");
+        WRITE_CODE(m_grain_seed, 16, "grain_seed");
+        WRITE_FLAG(1, "update_grain_flag");
+        WRITE_CODE(units_resolution_log2, 4, "apply_units_resolution_log2");
+        WRITE_CODE(horz_resolution, 12, "apply_horz_resolution");
+        WRITE_CODE(vert_resolution, 12, "apply_vert_resolution");
+        WRITE_FLAG(luma_only_flag, "luma_only_flag");
+        if (!luma_only_flag) {
+            WRITE_FLAG(subsamplingX, "subsampling_x");
+            WRITE_FLAG(subsamplingY, "subsampling_y");
+        }
+        WRITE_FLAG(sps.vuiParameters.videoSignalTypePresentFlag, "video_signal_characteristics_flag");
+        if (sps.vuiParameters.videoSignalTypePresentFlag)
         {
-            for (int i = 0; i < m_num_y_points; i++)
+            WRITE_CODE(m_bitDepth - 8, 3, "bit_depth_minus8");
+            WRITE_FLAG(sps.vuiParameters.colourDescriptionPresentFlag, "cicp_info_present_flag");
+            if (sps.vuiParameters.colourDescriptionPresentFlag)
             {
-                for (int j = 0; j < 2; j++)
-                {
-                    WRITE_CODE(m_scaling_points_yij, 8, "scaling_points_yij");
-                }
+                WRITE_CODE(sps.vuiParameters.colourPrimaries, 8, "colour_primaries");
+                WRITE_CODE(sps.vuiParameters.transferCharacteristics, 8, "transfer_characteristics");
+                WRITE_CODE(sps.vuiParameters.matrixCoefficients, 8, "matrix_coefficients");
+                WRITE_FLAG(sps.vuiParameters.videoFullRangeFlag, "video_full_range_flag");
             }
         }
-        WRITE_FLAG(m_num_cb_points == 0 && m_num_cr_points == 0, "luma_only_flag");
-        WRITE_FLAG(0, "chroma_scaling_from_luma");
-        WRITE_CODE(m_num_cb_points, 4, "num_cb_points");
-        if (m_num_cb_points)
-        {
-            for (int i = 0; i < m_num_cb_points; i++)
-            {
-                for (int j = 0; j < 2; j++)
-                {
-                    WRITE_CODE(m_scaling_points_cbij, 8, "scaling_points_cbij");
-                }
+        WRITE_FLAG(predict_scaling_flag, "predict_scaling_flag");
+        if (predict_scaling_flag) {
+            WRITE_FLAG(predict_y_scaling_flag, "predict_y_scaling_flag");
+        }
+
+        WRITE_CODE(m_num_y_points, 4, "num_y_points");
+        if (m_num_y_points) {
+            WRITE_CODE(point_y_value_increment_bits - 1, 3, "point_y_value_increment_bits_minus1");
+            int bitsIncr = point_y_value_increment_bits;
+            WRITE_CODE(point_y_scaling_bits - 5, 2, "point_y_scaling_bits_minus5");
+            int bitsScal = point_y_scaling_bits;
+            for (int i = 0; i < m_num_y_points; i++) {
+                if (i)
+                    WRITE_CODE(m_scaling_points_yi0 - m_scaling_points_yi - 10, bitsIncr, "point_y_value_incrementi");
+                else
+                    WRITE_CODE(m_scaling_points_yi0, bitsIncr, "point_y_value_incrementi");
+                WRITE_CODE(m_scaling_points_yi1, bitsScal, "point_y_scalingi");
             }
         }
-        WRITE_CODE(m_num_cr_points, 4, "num_cr_points");
-        if (m_num_cr_points)
-        {
-            for (int i = 0; i < m_num_cr_points; i++)
-            {
-                for (int j = 0; j < 2; j++)
-                {
-                    WRITE_CODE(m_scaling_points_crij, 8, "scaling_points_crij");
+
+        if (!luma_only_flag)
+            WRITE_FLAG(m_chroma_scaling_from_luma, "chroma_scaling_from_luma");
+
+        if (luma_only_flag || m_chroma_scaling_from_luma) {
+            m_num_cb_points = 0;
+            m_num_cr_points = 0;
+        }
+        else {
+            if (predict_scaling_flag)
+                WRITE_FLAG(predict_cb_scaling_flag, "predict_cb_scaling_flag");
+
+            WRITE_CODE(m_num_cb_points, 4, "num_cb_points");
+            if (m_num_cb_points) {
+                WRITE_CODE(point_cb_value_increment_bits - 1, 3, "point_cb_value_increment_bits_minus1");
+                int bitsIncr = point_cb_value_increment_bits;
+                WRITE_CODE(point_cb_scaling_bits - 5, 2, "point_cb_scaling_bits_minus5");
+                int bitsScal = point_cb_scaling_bits;
+                WRITE_CODE(cb_scaling_offset, 8, "cb_scaling_offset");
+
+                for (int i = 0; i < m_num_cb_points; i++) {
+                    if (i)
+                        WRITE_CODE(m_scaling_points_cbi0 - m_scaling_points_cbi - 10, bitsIncr, "point_cb_value_incrementi");
+                    else
+                        WRITE_CODE(m_scaling_points_cbi0, bitsIncr, "point_cb_value_incrementi");
+                    WRITE_CODE(m_scaling_points_cbi1, bitsScal, "point_cb_scalingi");
+                }
+            }
+
+            if (predict_scaling_flag)
+                WRITE_FLAG(predict_cr_scaling_flag, "predict_cr_scaling_flag");
+
+            WRITE_CODE(m_num_cr_points, 4, "num_cr_points");
+            if (m_num_cr_points) {
+                WRITE_CODE(point_cr_value_increment_bits - 1, 3, "point_cr_value_increment_bits_minus1");
+                int bitsIncr = point_cr_value_increment_bits;
+                WRITE_CODE(point_cr_scaling_bits - 5, 2, "point_cr_scaling_bits_minus5");
+                int bitsScal = point_cr_scaling_bits;
+                WRITE_CODE(cr_scaling_offset, 8, "cr_scaling_offset");
+
+                for (int i = 0; i < m_num_cr_points; i++) {
+                    if (i)
+                        WRITE_CODE(m_scaling_points_cri0 - m_scaling_points_cri - 10, bitsIncr, "point_cr_value_incrementi");
+                    else
+                        WRITE_CODE(m_scaling_points_cri0, bitsIncr, "point_cr_value_incrementi");
+                    WRITE_CODE(m_scaling_points_cri1, bitsScal, "point_cr_scalingi");
                 }
             }
         }
-        WRITE_CODE(m_scaling_shift - 8, 2, "scaling_shift");
+
+        WRITE_CODE(m_scaling_shift - 8, 2, "grain_scaling_minus8");
         WRITE_CODE(m_ar_coeff_lag, 2, "ar_coeff_lag");
-        if (m_num_y_points)
+        int bits_per_ar_coef = 8;
+        if (m_num_y_points || predict_y_scaling_flag)
         {
+            WRITE_CODE(bits_per_ar_coef - 5, 2, "bits_per_ar_coeff_y_minus5");
+
             for (int i = 0; i < 24; i++)
             {
-                WRITE_CODE(m_ar_coeffs_yi + 128, 8, "ar_coeff_yi");
+                WRITE_CODE(m_ar_coeffs_yi + (1 << (bits_per_ar_coef - 1)), bits_per_ar_coef, "ar_coeff_yi");
             }
         }
-        if (m_num_cb_points || m_chroma_scaling_from_luma)
+        if (m_num_cb_points || m_chroma_scaling_from_luma || predict_cb_scaling_flag)
         {
+            WRITE_CODE(bits_per_ar_coef - 5, 2, "bits_per_ar_coeff_cb_minus5");
+
             for (int i = 0; i < 25; i++)
             {
-                WRITE_CODE(m_ar_coeffs_cbi + 128, 8, "ar_coeff_cbi");
+                WRITE_CODE(m_ar_coeffs_cbi + (1 << (bits_per_ar_coef - 1)), bits_per_ar_coef, "ar_coeff_cbi");
             }
         }
-        if (m_num_cr_points || m_chroma_scaling_from_luma)
+        if (m_num_cr_points || m_chroma_scaling_from_luma || predict_cr_scaling_flag)
         {
+            WRITE_CODE(bits_per_ar_coef - 5, 2, "bits_per_ar_coeff_cr_minus5");
+
             for (int i = 0; i < 25; i++)
             {
-                WRITE_CODE(m_ar_coeffs_cri + 128, 8, "ar_coeff_cri");
+                WRITE_CODE(m_ar_coeffs_cri + (1 << (bits_per_ar_coef - 1)), bits_per_ar_coef, "ar_coeff_cri");
             }
         }
         WRITE_CODE(m_ar_coeff_shift - 6, 2, "ar_coeff_shift");
         WRITE_CODE(m_grain_scale_shift, 2, "grain_scale_shift");
-        if (m_num_cb_points)
+        if (m_num_cb_points && !predict_cb_scaling_flag)
         {
             WRITE_CODE(m_cb_mult, 8, "cb_mult");
             WRITE_CODE(m_cb_luma_mult, 8, "cb_luma_mult");
             WRITE_CODE(m_cb_offset, 9, "cb_offset");
         }
-        if (m_num_cr_points)
+        if (m_num_cr_points && !predict_cr_scaling_flag)
         {
             WRITE_CODE(m_cr_mult, 8, "cr_mult");
             WRITE_CODE(m_cr_luma_mult, 8, "cr_luma_mult");

x265-4.1.tar/source/encoder/slicetype.cpp -> x265-4.2.tar/source/encoder/slicetype.cpp Changed

@@ -937,7 +937,7 @@
     if (mindenom > 0 && !(minscale & 1))
     {
         unsigned long idx;
-        CTZ(idx, minscale);
+        BSF(idx, minscale);
         int shift = X265_MIN((int)idx, mindenom);
         mindenom -= shift;
         minscale >>= shift;
@@ -1284,7 +1284,10 @@
     slicetypeDecide();
 
     m_inputLock.acquire();
-    if (m_outputSignalRequired)
+    m_sliceTypeBusy = false;
+    m_helpWanted = true;
+
+    if (m_outputSignalRequired && m_outputQueue.size())
     {
         m_outputSignal.trigger();
         m_outputSignalRequired = false;
@@ -1473,7 +1476,6 @@
     uint64_t picTotVariance = 0;
     uint32_t variance;
 
-    uint64_t blockXY = 0;
     pixel* src = curFrame->m_fencPic->m_picOrg0;
 
     for (int blockY = 0; blockY < maxRow; blockY += 8)
@@ -1489,7 +1491,6 @@
                 blockOffsetLuma, 0);
 
             rowVariance += variance;
-            blockXY++;
         }
         picTotVariance += (uint16_t)(rowVariance / maxCol);
     }
@@ -1520,7 +1521,6 @@
                 blockOffsetChroma, 1);
 
             rowVariance += variance;
-            blockXY++;
         }
         picTotVariance += (uint16_t)(rowVariance / maxColChroma);
     }
@@ -1544,7 +1544,6 @@
                 blockOffsetChroma, 2);
 
             rowVariance += variance;
-            blockXY++;
         }
         picTotVariance += (uint16_t)(rowVariance / maxColChroma);
     }
@@ -1566,13 +1565,18 @@
 
 {
     *sum = 0;
+    uint8_t shift = X265_DEPTH - 8;
 
     for (uint32_t verticalIdx = 0; verticalIdx < inputHeight; verticalIdx += dsFactor)
     {
         for (uint32_t horizontalIdx = 0; horizontalIdx < inputWidth; horizontalIdx += dsFactor)
         {
-            ++(histograminputSrchorizontalIdx);
-            *sum += inputSrchorizontalIdx;
+            pixel val = inputSrchorizontalIdx >> shift;
+#if HIGH_BIT_DEPTH
+            X265_CHECK(val < HISTOGRAM_NUMBER_OF_BINS, "Pixel value out of allocated histogram range. This will lead to memory corruption.\n");
+#endif
+            ++(histogramval);
+            *sum += val;
         }
         inputSrc += (stride << (dsFactor >> 1));
     }
@@ -2759,27 +2763,33 @@
             break;
     }
 
-    if (!framecnt)
+    if (!framecnt && m_param->analysisLoadReuseLevel != 1)
     {
         if (m_param->rc.cuTree)
             cuTree(frames, 0, bKeyframe);
         return;
     }
-    framesframecnt + 1 = NULL;
 
-    if (m_param->bResetZoneConfig)
+    if (framecnt)
     {
-        for (int i = 0; i < m_param->rc.zonefileCount; i++)
+
+        framesframecnt + 1 = NULL;
+
+        if (m_param->bResetZoneConfig)
         {
-            int curZoneStart = m_param->rc.zonesi.startFrame, nextZoneStart = 0;
-            curZoneStart += curZoneStart ? m_param->rc.zonesi.zoneParam->radl : 0;
-            nextZoneStart += (i + 1 < m_param->rc.zonefileCount) ? m_param->rc.zonesi + 1.startFrame + m_param->rc.zonesi + 1.zoneParam->radl : m_param->totalFrames;
-            if (curZoneStart <= frames0->frameNum && nextZoneStart > frames0->frameNum)
-                m_param->keyframeMax = nextZoneStart - curZoneStart;
-            if (m_param->rc.zonesm_param->rc.zonefileCount - 1.startFrame <= frames0->frameNum && nextZoneStart == 0)
-                m_param->keyframeMax = m_param->rc.zones0.keyframeMax;
+            for (int i = 0; i < m_param->rc.zonefileCount; i++)
+            {
+                int curZoneStart = m_param->rc.zonesi.startFrame, nextZoneStart = 0;
+                curZoneStart += curZoneStart ? m_param->rc.zonesi.zoneParam->radl : 0;
+                nextZoneStart += (i + 1 < m_param->rc.zonefileCount) ? m_param->rc.zonesi + 1.startFrame + m_param->rc.zonesi + 1.zoneParam->radl : m_param->totalFrames;
+                if (curZoneStart <= frames0->frameNum && nextZoneStart > frames0->frameNum)
+                    m_param->keyframeMax = nextZoneStart - curZoneStart;
+                if (m_param->rc.zonesm_param->rc.zonefileCount - 1.startFrame <= frames0->frameNum && nextZoneStart == 0)
+                    m_param->keyframeMax = m_param->rc.zones0.keyframeMax;
+            }
         }
     }
+
     int keylimit = m_param->keyframeMax;
     if (frames0->frameNum < m_param->chunkEnd)
     {
@@ -2798,266 +2808,281 @@
         keyintLimit = keyFrameLimit;
 
     origNumFrames = numFrames = m_param->bIntraRefresh ? framecnt : X265_MIN(framecnt, keyintLimit);
-    if (bIsVbvLookahead)
-        numFrames = framecnt;
-    else if (m_param->bOpenGOP && numFrames < framecnt)
-        numFrames++;
-    else if (numFrames == 0)
-    {
-        frames1->sliceType = X265_TYPE_I;
-        return;
-    }
 
-    if (m_bBatchMotionSearch)
+    if (framecnt)
     {
-        /* pre-calculate all motion searches, using many worker threads */
-        CostEstimateGroup estGroup(*this, frames);
-        for (int b = 2; b < numFrames; b++)
+        if (bIsVbvLookahead)
+            numFrames = framecnt;
+        else if (m_param->bOpenGOP && numFrames < framecnt)
+            numFrames++;
+        else if (numFrames == 0)
         {
-            for (int i = 1; i <= m_param->bframes + 1; i++)
-            {
-                int p0 = b - i;
-                if (p0 < 0)
-                    continue;
-
-                /* Skip search if already done */
-                if (framesb->lowresMvs0i0.x != 0x7FFF)
-                    continue;
-
-                /* perform search to p1 at same distance, if possible */
-                int p1 = b + i;
-                if (p1 >= numFrames || framesb->lowresMvs1i0.x != 0x7FFF)
-                    p1 = b;
-
-                estGroup.add(p0, p1, b);
-            }
+            frames1->sliceType = X265_TYPE_I;
+            return;
         }
-        /* auto-disable after the first batch if pool is small */
-        m_bBatchMotionSearch &= m_pool->m_numWorkers >= 4;
-        estGroup.finishBatch();
 
-        if (m_bBatchFrameCosts)
+        if (m_bBatchMotionSearch)
         {
-            /* pre-calculate all frame cost estimates, using many worker threads */
+            /* pre-calculate all motion searches, using many worker threads */
+            CostEstimateGroup estGroup(*this, frames);
             for (int b = 2; b < numFrames; b++)
             {
                 for (int i = 1; i <= m_param->bframes + 1; i++)
                 {
-                    if (b < i)
+                    int p0 = b - i;
+                    if (p0 < 0)
                         continue;
 
-                    /* only measure frame cost in this pass if motion searches
-                     * are already done */
-                    if (framesb->lowresMvs0i0.x == 0x7FFF)
+                    /* Skip search if already done */
+                    if (framesb->lowresMvs0i0.x != 0x7FFF)
                         continue;
 
-                    int p0 = b - i;
+                    /* perform search to p1 at same distance, if possible */
+                    int p1 = b + i;
+                    if (p1 >= numFrames || framesb->lowresMvs1i0.x != 0x7FFF)
+                        p1 = b;
 
-                    for (int j = 0; j <= m_param->bframes; j++)
-                    {
-                        int p1 = b + j;
-                        if (p1 >= numFrames)
-                            break;
+                    estGroup.add(p0, p1, b);
+                }
+            }
+            /* auto-disable after the first batch if pool is small */
+            m_bBatchMotionSearch &= m_pool->m_numWorkers >= 4;
+            estGroup.finishBatch();
 
-                        /* ensure P1 search is done */
-                        if (j && framesb->lowresMvs1j0.x == 0x7FFF)
+            if (m_bBatchFrameCosts)
+            {
+                /* pre-calculate all frame cost estimates, using many worker threads */
+                for (int b = 2; b < numFrames; b++)
+                {
+                    for (int i = 1; i <= m_param->bframes + 1; i++)
+                    {
+                        if (b < i)
                             continue;
 
-                        /* ensure frame cost is not done */
-                        if (framesb->costEstij >= 0)
+                        /* only measure frame cost in this pass if motion searches
+                         * are already done */
+                        if (framesb->lowresMvs0i0.x == 0x7FFF)
                             continue;
 
-                        estGroup.add(p0, p1, b);
+                        int p0 = b - i;
+
+                        for (int j = 0; j <= m_param->bframes; j++)
+                        {
+                            int p1 = b + j;
+                            if (p1 >= numFrames)
+                                break;
+
+                            /* ensure P1 search is done */
+                            if (j && framesb->lowresMvs1j0.x == 0x7FFF)
+                                continue;
+
+                            /* ensure frame cost is not done */
+                            if (framesb->costEstij >= 0)
+                                continue;
+
+                            estGroup.add(p0, p1, b);
+                        }
                     }
                 }
-            }
 
-            /* auto-disable after the first batch if the pool is not large */
-            m_bBatchFrameCosts &= m_pool->m_numWorkers > 12;
-            estGroup.finishBatch();
+                /* auto-disable after the first batch if the pool is not large */
+                m_bBatchFrameCosts &= m_pool->m_numWorkers > 12;
+                estGroup.finishBatch();
+            }
         }
-    }
 
-    int numBFrames = 0;
-    int numAnalyzed = numFrames;
-    bool isScenecut = false;
+        int numBFrames = 0;
+        int numAnalyzed = numFrames;
+        bool isScenecut = false;
 
-    if (m_param->bHistBasedSceneCut)
-        isScenecut = histBasedScenecut(frames, 0, 1, origNumFrames);
-    else
-        isScenecut = scenecut(frames, 0, 1, true, origNumFrames);
+        if (m_param->bHistBasedSceneCut)
+            isScenecut = histBasedScenecut(frames, 0, 1, origNumFrames);
+        else
+            isScenecut = scenecut(frames, 0, 1, true, origNumFrames);
 
-    /* When scenecut threshold is set, use scenecut detection for I frame placements */
-    if (m_param->scenecutThreshold && isScenecut)
-    {
-        frames1->sliceType = X265_TYPE_I;
-        return;
-    }
-    if (m_param->gopLookahead && (keyFrameLimit >= 0) && (keyFrameLimit <= m_param->bframes + 1))
-    {
-        bool sceneTransition = m_isSceneTransition;
-        m_extendGopBoundary = false;
-        for (int i = m_param->bframes + 1; i < origNumFrames; i += m_param->bframes + 1)
+        /* When scenecut threshold is set, use scenecut detection for I frame placements */
+        if (m_param->scenecutThreshold && isScenecut)
         {
-            scenecut(frames, i, i + 1, true, origNumFrames);
-
-            for (int j = i + 1; j <= X265_MIN(i + m_param->bframes + 1, origNumFrames); j++)
+            frames1->sliceType = X265_TYPE_I;
+            return;
+        }
+        if (m_param->gopLookahead && (keyFrameLimit >= 0) && (keyFrameLimit <= m_param->bframes + 1))
+        {
+            bool sceneTransition = m_isSceneTransition;
+            m_extendGopBoundary = false;
+            for (int i = m_param->bframes + 1; i < origNumFrames; i += m_param->bframes + 1)
             {
-                if (framesj->bScenecut && scenecutInternal(frames, j - 1, j, true))
+                scenecut(frames, i, i + 1, true, origNumFrames);
+
+                for (int j = i + 1; j <= X265_MIN(i + m_param->bframes + 1, origNumFrames); j++)
                 {
-                    m_extendGopBoundary = true;
-                    break;
+                    if (framesj->bScenecut && scenecutInternal(frames, j - 1, j, true))
+                    {
+                        m_extendGopBoundary = true;
+                        break;
+                    }
                 }
+                if (m_extendGopBoundary)
+                    break;
             }
-            if (m_extendGopBoundary)
-                break;
+            m_isSceneTransition = sceneTransition;
         }
-        m_isSceneTransition = sceneTransition;
-    }
-    if (m_param->bframes)
-    {
-        if (m_param->bFrameAdaptive == X265_B_ADAPT_TRELLIS)
+        if (m_param->bframes)
         {
-            if (numFrames > 1)
+            if (m_param->bFrameAdaptive == X265_B_ADAPT_TRELLIS)
             {
-                char best_pathsX265_BFRAME_MAX + 1X265_LOOKAHEAD_MAX + 1 = { "", "P" };
-                int best_path_index = numFrames % (X265_BFRAME_MAX + 1);
+                if (numFrames > 1)
+                {
+                    char best_pathsX265_BFRAME_MAX + 1X265_LOOKAHEAD_MAX + 1 = { "", "P" };
+                    int best_path_index = numFrames % (X265_BFRAME_MAX + 1);
 
-                /* Perform the frame type analysis. */
-                for (int j = 2; j <= numFrames; j++)
-                    slicetypePath(frames, j, best_paths);
+                    /* Perform the frame type analysis. */
+                    for (int j = 2; j <= numFrames; j++)
+                        slicetypePath(frames, j, best_paths);
 
-                numBFrames = (int)strspn(best_pathsbest_path_index, "B");
+                    numBFrames = (int)strspn(best_pathsbest_path_index, "B");
 
-                /* Load the results of the analysis into the frame types. */
-                for (int j = 1; j < numFrames; j++)
-                    framesj->sliceType = best_pathsbest_path_indexj - 1 == 'B' ? X265_TYPE_B : X265_TYPE_P;
+                    /* Load the results of the analysis into the frame types. */
+                    for (int j = 1; j < numFrames; j++)
+                        framesj->sliceType = best_pathsbest_path_indexj - 1 == 'B' ? X265_TYPE_B : X265_TYPE_P;
+                }
+                framesnumFrames->sliceType = X265_TYPE_P;
             }
-            framesnumFrames->sliceType = X265_TYPE_P;
-        }
-        else if (m_param->bFrameAdaptive == X265_B_ADAPT_FAST)
-        {
-            CostEstimateGroup estGroup(*this, frames);
+            else if (m_param->bFrameAdaptive == X265_B_ADAPT_FAST)
+            {
+                CostEstimateGroup estGroup(*this, frames);
 
-            int64_t cost1p0, cost2p0, cost1b1, cost2p1;
+                int64_t cost1p0, cost2p0, cost1b1, cost2p1;
 
-            for (int i = 0; i <= numFrames - 2; )
-            {
-                cost2p1 = estGroup.singleCost(i + 0, i + 2, i + 2, true);
-                if (framesi + 2->intraMbs2 > cuCount / 2)
+                for (int i = 0; i <= numFrames - 2; )
                 {
-                    framesi + 1->sliceType = X265_TYPE_P;
-                    framesi + 2->sliceType = X265_TYPE_P;
-                    i += 2;
-                    continue;
-                }
+                    cost2p1 = estGroup.singleCost(i + 0, i + 2, i + 2, true);
+                    if (framesi + 2->intraMbs2 > cuCount / 2)
+                    {
+                        framesi + 1->sliceType = X265_TYPE_P;
+                        framesi + 2->sliceType = X265_TYPE_P;
+                        i += 2;
+                        continue;
+                    }
 
-                cost1b1 = estGroup.singleCost(i + 0, i + 2, i + 1);
-                cost1p0 = estGroup.singleCost(i + 0, i + 1, i + 1);
-                cost2p0 = estGroup.singleCost(i + 1, i + 2, i + 2);
+                    cost1b1 = estGroup.singleCost(i + 0, i + 2, i + 1);
+                    cost1p0 = estGroup.singleCost(i + 0, i + 1, i + 1);
+                    cost2p0 = estGroup.singleCost(i + 1, i + 2, i + 2);
 
-                if (cost1p0 + cost2p0 < cost1b1 + cost2p1)
-                {
-                    framesi + 1->sliceType = X265_TYPE_P;
-                    i += 1;
-                    continue;
-                }
+                    if (cost1p0 + cost2p0 < cost1b1 + cost2p1)
+                    {
+                        framesi + 1->sliceType = X265_TYPE_P;
+                        i += 1;
+                        continue;
+                    }
 
-// arbitrary and untuned
+                    // arbitrary and untuned
 #define INTER_THRESH 300
 #define P_SENS_BIAS (50 - m_param->bFrameBias)
-                framesi + 1->sliceType = X265_TYPE_B;
+                    framesi + 1->sliceType = X265_TYPE_B;
+
+                    int j;
+                    for (j = i + 2; j <= X265_MIN(i + m_param->bframes, numFrames - 1); j++)
+                    {
+                        int64_t pthresh = X265_MAX(INTER_THRESH - P_SENS_BIAS * (j - i - 1), INTER_THRESH / 10);
+                        int64_t pcost = estGroup.singleCost(i + 0, j + 1, j + 1, true);
+                        if (pcost > pthresh * cuCount || framesj + 1->intraMbsj - i + 1 > cuCount / 3)
+                            break;
+                        framesj->sliceType = X265_TYPE_B;
+                    }
+
+                    framesj->sliceType = X265_TYPE_P;
+                    i = j;
+                }
+                framesnumFrames->sliceType = X265_TYPE_P;
+                numBFrames = 0;
+                while (numBFrames < numFrames && framesnumBFrames + 1->sliceType == X265_TYPE_B)
+                    numBFrames++;
+            }
+            else
+            {
+                numBFrames = X265_MIN(numFrames - 1, m_param->bframes);
+                for (int j = 1; j < numFrames; j++)
+                    framesj->sliceType = (j % (numBFrames + 1)) ? X265_TYPE_B : X265_TYPE_P;
+
+                framesnumFrames->sliceType = X265_TYPE_P;
+            }
 
-                int j;
-                for (j = i + 2; j <= X265_MIN(i + m_param->bframes, numFrames - 1); j++)
+            int zoneRadl = m_param->rc.zonefileCount && m_param->bResetZoneConfig ? m_param->rc.zones->zoneParam->radl : 0;
+            bool bForceRADL = zoneRadl || (m_param->radl && (m_param->keyframeMax == m_param->keyframeMin));
+            bool bLastMiniGop = (framecnt >= m_param->bframes + 1) ? false : true;
+            int radl = m_param->radl ? m_param->radl : zoneRadl;
+            int preRADL = m_lastKeyframe + m_param->keyframeMax - radl - 1; /*Frame preceeding RADL in POC order*/
+            if (bForceRADL && (frames0->frameNum == preRADL) && !bLastMiniGop)
+            {
+                int j = 1;
+                numBFrames = m_param->radl ? m_param->radl : zoneRadl;
+                for (; j <= numBFrames; j++)
+                    framesj->sliceType = X265_TYPE_B;
+                framesj->sliceType = X265_TYPE_I;
+            }
+            else /* Check scenecut and RADL on the first minigop. */
+            {
+                for (int j = 1; j < numBFrames + 1; j++)
                 {
-                    int64_t pthresh = X265_MAX(INTER_THRESH - P_SENS_BIAS * (j - i - 1), INTER_THRESH / 10);
-                    int64_t pcost = estGroup.singleCost(i + 0, j + 1, j + 1, true);
-                    if (pcost > pthresh * cuCount || framesj + 1->intraMbsj - i + 1 > cuCount / 3)
+                    if (scenecut(frames, j, j + 1, false, origNumFrames) ||
+                        (bForceRADL && (framesj->frameNum == preRADL)))
+                    {
+                        framesj->sliceType = X265_TYPE_P;
+                        numAnalyzed = j;
                         break;
-                    framesj->sliceType = X265_TYPE_B;
+                    }
                 }
-
-                framesj->sliceType = X265_TYPE_P;
-                i = j;
             }
-            framesnumFrames->sliceType = X265_TYPE_P;
-            numBFrames = 0;
-            while (numBFrames < numFrames && framesnumBFrames + 1->sliceType == X265_TYPE_B)
-                numBFrames++;
+            resetStart = bKeyframe ? 1 : X265_MIN(numBFrames + 2, numAnalyzed + 1);
         }
         else
         {
-            numBFrames = X265_MIN(numFrames - 1, m_param->bframes);
-            for (int j = 1; j < numFrames; j++)
-                framesj->sliceType = (j % (numBFrames + 1)) ? X265_TYPE_B : X265_TYPE_P;
+            for (int j = 1; j <= numFrames; j++)
+                framesj->sliceType = X265_TYPE_P;
 
-            framesnumFrames->sliceType = X265_TYPE_P;
+            resetStart = bKeyframe ? 1 : 2;
         }
+        if (m_param->bAQMotion)
+            aqMotion(frames, bKeyframe);
 
-        int zoneRadl = m_param->rc.zonefileCount && m_param->bResetZoneConfig ? m_param->rc.zones->zoneParam->radl : 0;
-        bool bForceRADL = zoneRadl || (m_param->radl && (m_param->keyframeMax == m_param->keyframeMin));
-        bool bLastMiniGop = (framecnt >= m_param->bframes + 1) ? false : true;
-        int radl = m_param->radl ? m_param->radl : zoneRadl;
-        int preRADL = m_lastKeyframe + m_param->keyframeMax - radl - 1; /*Frame preceeding RADL in POC order*/
-        if (bForceRADL && (frames0->frameNum == preRADL) && !bLastMiniGop)
-        {
-            int j = 1;
-            numBFrames = m_param->radl ? m_param->radl : zoneRadl;
-            for (; j <= numBFrames; j++)
-                framesj->sliceType = X265_TYPE_B;
-            framesj->sliceType = X265_TYPE_I;
-        }
-        else /* Check scenecut and RADL on the first minigop. */
-        {
-            for (int j = 1; j < numBFrames + 1; j++)
-            {
-                if (scenecut(frames, j, j + 1, false, origNumFrames) ||
-                    (bForceRADL && (framesj->frameNum == preRADL)))
-                {
-                    framesj->sliceType = X265_TYPE_P;
-                    numAnalyzed = j;
-                    break;
-                }
-            }
-        }
-        resetStart = bKeyframe ? 1 : X265_MIN(numBFrames + 2, numAnalyzed + 1);
-    }
-    else
-    {
-        for (int j = 1; j <= numFrames; j++)
-            framesj->sliceType = X265_TYPE_P;
+        if (m_param->rc.cuTree)
+            cuTree(frames, X265_MIN(numFrames, m_param->keyframeMax), bKeyframe);
 
-        resetStart = bKeyframe ? 1 : 2;
-    }
-    if (m_param->bAQMotion)
-        aqMotion(frames, bKeyframe);
+        if (m_param->gopLookahead && (keyFrameLimit >= 0) && (keyFrameLimit <= m_param->bframes + 1) && !m_extendGopBoundary)
+            keyintLimit = keyFrameLimit;
 
-    if (m_param->rc.cuTree)
-        cuTree(frames, X265_MIN(numFrames, m_param->keyframeMax), bKeyframe);
+        if (!m_param->bIntraRefresh)
+            for (int j = keyintLimit + 1; j <= numFrames; j += m_param->keyframeMax)
+            {
+                framesj->sliceType = X265_TYPE_I;
+                resetStart = X265_MIN(resetStart, j + 1);
+            }
 
-    if (m_param->gopLookahead && (keyFrameLimit >= 0) && (keyFrameLimit <= m_param->bframes + 1) && !m_extendGopBoundary)
-        keyintLimit = keyFrameLimit;
+        if (bIsVbvLookahead)
+            vbvLookahead(frames, numFrames, bKeyframe);
+        int maxp1 = X265_MIN(m_param->bframes + 1, origNumFrames);
 
-    if (!m_param->bIntraRefresh)
-        for (int j = keyintLimit + 1; j <= numFrames; j += m_param->keyframeMax)
+        /* Restore frame types for all frames that haven't actually been decided yet. */
+        for (int j = resetStart; j <= numFrames; j++)
         {
-            framesj->sliceType = X265_TYPE_I;
-            resetStart = X265_MIN(resetStart, j + 1);
+            framesj->sliceType = X265_TYPE_AUTO;
+            /* If any frame marked as scenecut is being restarted for sliceDecision,
+             * undo scene Transition flag */
+            if (j <= maxp1 && framesj->bScenecut && m_isSceneTransition)
+                m_isSceneTransition = false;
         }
+    }
 
-    if (bIsVbvLookahead)
-        vbvLookahead(frames, numFrames, bKeyframe);
-    int maxp1 = X265_MIN(m_param->bframes + 1, origNumFrames);
-
-    /* Restore frame types for all frames that haven't actually been decided yet. */
-    for (int j = resetStart; j <= numFrames; j++)
+    if (m_param->rc.cuTree && !framecnt)
     {
-        framesj->sliceType = X265_TYPE_AUTO;
-        /* If any frame marked as scenecut is being restarted for sliceDecision, 
-         * undo scene Transition flag */
-        if (j <= maxp1 && framesj->bScenecut && m_isSceneTransition)
-            m_isSceneTransition = false;
+        for (framecnt = 0; framecnt < maxSearch; framecnt++)
+        {
+            Lowres* shhh = framesframecnt + 1;
+            if (!shhh)
+                break;
+        }
+        cuTree(frames, X265_MIN(framecnt, m_param->keyframeMax), bKeyframe);
     }
 }

x265-4.2.tar/source/encoder/threadedme.cpp Added

@@ -0,0 +1,260 @@
+/*****************************************************************************
+ * Copyright (C) 2013-2025 MulticoreWare, Inc
+ *
+ * Authors: Shashank Pathipati <shashank.pathipati@multicorewareinc.com>
+ *          Somu Vineela <somu@mutlicorewareinc.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#include "threadedme.h"
+#include "encoder.h"
+#include "frameencoder.h"
+
+#include <iostream>
+#include <sstream>
+
+namespace X265_NS {
+int g_puStartIdx1288 = {0};
+
+bool ThreadedME::create()
+{
+    m_active = true;
+    m_tldCount = m_pool->m_numWorkers;
+    m_tld = new ThreadLocalDatam_tldCount;
+    for (int i = 0; i < m_tldCount; i++)
+    {
+        m_tldi.analysis.initSearch(*m_param, m_enc.m_scalingList);
+        m_tldi.analysis.create(m_tld);
+    }
+
+    initPuStartIdx();
+
+    /* start sequence at zero */
+    m_enqueueSeq = 0ULL;
+
+    return true;
+}
+
+void ThreadedME::initPuStartIdx()
+{
+    int startIdx = 0;
+    uint32_t ctuSize = m_param->maxCUSize;
+
+    for (uint32_t puIdx = 0; puIdx < MAX_NUM_PU_SIZES; ++puIdx)
+    {
+        const PUBlock& pu = g_puLookuppuIdx;
+
+        if (pu.width > ctuSize || pu.height > ctuSize)
+            continue;
+
+        int indexWidth = pu.isAmp ? X265_MAX(pu.width, pu.height) : pu.width;
+        int indexHeight = pu.isAmp ? indexWidth : pu.height;
+
+        int numPUs = (ctuSize / indexWidth) * (ctuSize / indexHeight);
+        int partIdx = static_cast<int>(pu.partsize);
+
+        g_puStartIdxpu.width + pu.heightpartIdx = startIdx;
+
+        startIdx += pu.isAmp ? 2 * numPUs : numPUs;
+    }
+}
+
+void ThreadedME::enqueueCTUBlock(int row, int col, int width, int height, int layer, FrameEncoder* frameEnc)
+{
+    frameEnc->m_tmeTasksLock.acquire();
+
+    Frame* frame = frameEnc->m_framelayer;
+
+    CTUTask task;
+    task.seq = ATOMIC_ADD(&m_enqueueSeq, 1ULL);
+    task.row = row;
+    task.col = col;
+    task.width = width;
+    task.height = height;
+    task.layer = layer;
+
+    task.frame = frame;
+    task.frameEnc = frameEnc;
+
+    frameEnc->m_tmeTasks.push(task);
+    frameEnc->m_tmeTasksLock.release();
+
+    m_taskEvent.trigger();
+}
+
+void ThreadedME::enqueueReadyRows(int row, int layer, FrameEncoder* frameEnc)
+{
+    int bufRow = X265_MIN(row + m_param->tmeNumBufferRows, static_cast<int>(frameEnc->m_numRows));
+
+    for (int r = 0; r < bufRow; r++)
+    {
+        if (frameEnc->m_tmeDepsr.isQueued)
+            continue;
+
+        bool isInitialRow = r < m_param->tmeNumBufferRows;
+        bool isExternalDepResolved = frameEnc->m_tmeDepsr.external;
+
+        int prevRow = X265_MAX(0, r - m_param->tmeNumBufferRows);
+        bool isInternalDepResolved = frameEnc->m_tmeDepsprevRow.internal;
+
+        if ((isInitialRow && isExternalDepResolved) ||
+            (!isInitialRow && isExternalDepResolved && isInternalDepResolved))
+        {
+            int cols = static_cast<int>(frameEnc->m_numCols);
+            for (int c = 0; c < cols; c += m_param->tmeTaskBlockSize)
+            {
+                int blockWidth = X265_MIN(m_param->tmeTaskBlockSize, cols - c);
+                enqueueCTUBlock(r, c, blockWidth, 1, layer, frameEnc);
+            }
+            frameEnc->m_tmeDepsr.isQueued = true;
+        }
+    }
+}
+
+void ThreadedME::threadMain()
+{
+    while (m_active)
+    {
+        int newCTUsPushed = 0;
+
+        for (int i = 0; i < m_param->frameNumThreads; i++)
+        {
+            FrameEncoder* frameEnc = m_enc.m_frameEncoderi;
+            frameEnc->m_tmeTasksLock.acquire();
+
+            while (!frameEnc->m_tmeTasks.empty())
+            {
+                CTUTask task = frameEnc->m_tmeTasks.front();
+                frameEnc->m_tmeTasks.pop();
+
+                m_taskQueueLock.acquire();
+                m_taskQueue.push(task);
+                m_taskQueueLock.release();
+
+                newCTUsPushed++;
+                tryWakeOne();
+            }
+
+            frameEnc->m_tmeTasksLock.release();
+        }
+
+        if (newCTUsPushed == 0)
+            m_taskEvent.wait();
+    }
+}
+
+void ThreadedME::findJob(int workerThreadId)
+{
+    m_taskQueueLock.acquire();
+    if (m_taskQueue.empty())
+    {
+        m_helpWanted = false;
+        m_taskQueueLock.release();
+        return;
+    }
+    
+    m_helpWanted = true;
+    int64_t stime = x265_mdate();
+
+#ifdef DETAILED_CU_STATS
+    ScopedElapsedTime tmeTime(m_tldworkerThreadId.analysis.m_statsm_jpId.tmeTime);
+    m_tldworkerThreadId.analysis.m_statsm_jpId.countTmeTasks++;
+#endif
+
+    CTUTask task = m_taskQueue.top();
+    m_taskQueue.pop();
+    m_taskQueueLock.release();
+
+    int numCols = (m_param->sourceWidth + m_param->maxCUSize - 1) / m_param->maxCUSize;
+    Frame* frame = task.frame;
+
+    for (int i = 0; i < task.height; i++)
+    {
+        for (int j = 0; j < task.width; j++)
+        {
+
+            int ctuAddr = (task.row + i) * numCols + (task.col + j);
+            CUData* ctu = frame->m_encData->getPicCTU(ctuAddr);
+            ctu->m_slice = frame->m_encData->m_slice;
+
+            task.ctu = ctu;
+            task.geom = &task.frameEnc->m_cuGeomstask.frameEnc->m_ctuGeomMapctuAddr;
+
+            frame->m_encData->m_cuStatctuAddr.baseQp = frame->m_encData->m_avgQpRc;
+            initCTU(*ctu, task.row + i, task.col + j, task);
+
+            task.frame->m_ctuMEFlagsctuAddr.set(0);
+            m_tldworkerThreadId.analysis.deriveMVsForCTU(*task.ctu, *task.geom, *frame);
+
+            task.frame->m_ctuMEFlagsctuAddr.set(1);
+        }
+    }
+
+    if (m_param->csvLogLevel >= 2)
+    {
+        int64_t etime = x265_mdate();
+        ATOMIC_ADD(&task.frameEnc->m_totalThreadedMETimetask.layer, etime - stime);
+    }
+
+    m_taskEvent.trigger();
+}
+
+
+void ThreadedME::stopJobs()
+{
+    this->m_active = false;
+    m_taskEvent.trigger();
+}
+
+void ThreadedME::destroy()
+{
+    for (int i = 0; i < m_tldCount; i++)
+        m_tldi.destroy();
+    delete m_tld;
+}
+
+void ThreadedME::collectStats()
+{
+#ifdef DETAILED_CU_STATS
+    for (int i = 0; i < m_tldCount; i++)
+        m_cuStats.accumulate(m_tldi.analysis.m_statsm_jpId, *m_param);
+#endif
+}
+
+void initCTU(CUData& ctu, int row, int col, CTUTask& task)
+{
+    Frame& frame = *task.frame;
+    FrameEncoder& frameEnc = *task.frameEnc;
+
+    int numRows = frameEnc.m_numRows;
+    int numCols = frameEnc.m_numCols;
+    Slice *slice = frame.m_encData->m_slice;
+    CTURow& ctuRow = frameEnc.m_rowsrow;
+
+    const uint32_t bFirstRowInSlice = ((row == 0) || (frameEnc.m_rowsrow - 1.sliceId != ctuRow.sliceId)) ? 1 : 0;
+    const uint32_t bLastRowInSlice = ((row == numRows - 1) || (frameEnc.m_rowsrow + 1.sliceId != ctuRow.sliceId)) ? 1 : 0;
+
+    const uint32_t bLastCuInSlice = (bLastRowInSlice & (col == numCols - 1)) ? 1 : 0;
+
+    int ctuAddr = (numCols * row) + col;
+
+    ctu.initCTU(frame, ctuAddr, slice->m_sliceQp, bFirstRowInSlice, bLastRowInSlice, bLastCuInSlice);
+}
+
+}
\ No newline at end of file

x265-4.2.tar/source/encoder/threadedme.h Added

@@ -0,0 +1,249 @@
+/*****************************************************************************
+ * Copyright (C) 2013-2025 MulticoreWare, Inc
+ *
+ * Authors: Shashank Pathipati <shashank.pathipati@multicorewareinc.com>
+ *          Somu Vineela <somu@mutlicorewareinc.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02111, USA.
+ *
+ * This program is also available under a commercial proprietary license.
+ * For more information, contact us at license @ x265.com.
+ *****************************************************************************/
+
+#ifndef THREADED_ME_H
+#define THREADED_ME_H
+
+#include "common.h"
+#include "threading.h"
+#include "threadpool.h"
+#include "cudata.h"
+#include "lowres.h"
+#include "frame.h"
+#include "analysis.h"
+#include "mv.h"
+
+#include <queue>
+#include <vector>
+#include <fstream>
+
+namespace X265_NS {
+
+extern int g_puStartIdx1288;
+
+class Encoder;
+class Analysis;
+class FrameEncoder;
+
+struct PUBlock {
+    uint32_t width;
+    uint32_t height;
+    PartSize partsize;
+    bool isAmp;
+};
+
+const PUBlock g_puLookupMAX_NUM_PU_SIZES = {
+    { 8,   4, SIZE_2NxN,  0 },
+    { 4,   8, SIZE_Nx2N,  0 },
+    { 8,   8, SIZE_2Nx2N, 0 },
+    { 16,  4, SIZE_2NxnU, 1 },
+    { 16, 12, SIZE_2NxnD, 1 },
+    { 4,  16, SIZE_nLx2N, 1 },
+    { 12, 16, SIZE_nRx2N, 1 },
+    { 16,  8, SIZE_2NxN,  0 },
+    { 8,  16, SIZE_Nx2N,  0 },
+    { 16, 16, SIZE_2Nx2N, 0 },
+    { 32,  8, SIZE_2NxnU, 1 },
+    { 32, 24, SIZE_2NxnD, 1 },
+    { 8,  32, SIZE_nLx2N, 1 },
+    { 24, 32, SIZE_nRx2N, 1 },
+    { 32, 16, SIZE_2NxN,  0 },
+    { 16, 32, SIZE_Nx2N,  0 },
+    { 32, 32, SIZE_2Nx2N, 0 },
+    { 64, 16, SIZE_2NxnU, 1 },
+    { 64, 48, SIZE_2NxnD, 1 },
+    { 16, 64, SIZE_nLx2N, 1 },
+    { 48, 64, SIZE_nRx2N, 1 },
+    { 64, 32, SIZE_2NxN,  0 },
+    { 32, 64, SIZE_Nx2N,  0 },
+    { 64, 64, SIZE_2Nx2N, 0 }
+};
+
+struct CTUTaskData
+{
+    CUData& ctuData;
+    CUGeom& ctuGeom;
+    Frame& frame;
+};
+
+struct CTUBlockTask
+{
+    int row;
+    int col;
+    int width;
+    int height;
+    Frame* frame;
+    class FrameEncoder* frameEnc;
+    unsigned long long seq; /* monotonic sequence to preserve enqueue order */
+};
+
+struct PUData
+{
+    PartSize part;
+    const CUGeom* cuGeom;
+    int puOffset;
+    int areaId;
+    int finalIdx;
+    int qp;
+};
+
+struct MEData
+{
+    MV       mv2;
+    MV       mvp2;
+    uint32_t mvCost2;
+    int      ref2;
+    int      bits;
+    uint32_t cost;
+};
+
+struct CTUTask
+{
+    uint64_t seq;
+    int row;
+    int col;
+    int width;
+    int height;
+    int layer;
+
+    CUData* ctu;
+    CUGeom* geom;
+    Frame* frame;
+    FrameEncoder* frameEnc;
+};
+
+
+struct CompareCTUTask {
+    bool operator()(const CTUTask& a, const CTUTask& b) const {
+        if (a.frame->m_poc == b.frame->m_poc)
+        {
+            int a_pos = a.row + a.col;
+            int b_pos = b.row + b.col;
+            if (a_pos != b_pos) return a_pos > b_pos;
+        }
+
+        /* Compare by sequence number to preserve FIFO enqueue order.
+         * priority_queue in C++ is a max-heap, so return true when a.seq > b.seq
+         * to make smaller seq (earlier enqueue) the top() element. */
+        return a.seq > b.seq;
+    }
+};
+
+/**
+ * @brief Threaded motion-estimation module that schedules CTU blocks across worker threads.
+ *
+ * Owns per-worker analysis state (ThreadLocalData), manages the CTU task queues,
+ * and exposes a JobProvider interface for the thread pool to execute MVP
+ * derivation and ME searches in parallel.
+ */
+class ThreadedME: public JobProvider, public Thread
+{
+public:
+    x265_param*             m_param;
+    Encoder&                m_enc;
+
+    std::priority_queue<CTUTask, std::vector<CTUTask>, CompareCTUTask>  m_taskQueue;
+    Lock                    m_taskQueueLock;
+    Event                   m_taskEvent;
+
+    volatile bool           m_active;
+    unsigned long long      m_enqueueSeq;
+
+    ThreadLocalData*        m_tld;
+    int                     m_tldCount;
+
+#ifdef DETAILED_CU_STATS
+    CUStats                 m_cuStats;
+#endif
+
+    /**
+     * @brief Construct the ThreadedME manager; call create() before use.
+     */
+    ThreadedME(x265_param* param, Encoder& enc): m_param(param), m_enc(enc) {};
+    
+    /**
+     * @brief Creates threadpool, thread local data and registers itself as a job provider
+     */
+    bool create();
+
+    /**
+     * @brief Initialize lookup table used to index PU offsets for all valid CTU sizes.
+     */
+    void initPuStartIdx();
+
+    /**
+     * @brief Enqueue a block of CTUs for motion estimation.
+     *
+     * Blocks are queued per FrameEncoder and later moved into the global
+     * priority queue consumed by worker threads.
+     */
+    void enqueueCTUBlock(int row, int col, int width, int height, int layer, FrameEncoder* frameEnc);
+
+    /**
+     * @brief Inspect dependency state and enqueue newly-unblocked CTU rows.
+     *
+     * Uses external (row-level) and internal (buffered-row) dependencies to
+     * decide when a row can be split into CTU block tasks.
+     */
+    void enqueueReadyRows(int row, int layer, FrameEncoder* frameEnc);
+
+    /**
+     * @brief Main dispatcher thread that transfers per-frame tasks into the global queue.
+     */
+    void threadMain();
+
+    /**
+     * @brief Dequeue a CTU task, derive MVs, and run ME over all supported PU shapes.
+     *
+     * Called by worker threads via JobProvider; processes an entire CTU block.
+     */
+    void findJob(int workerThreadId);
+
+    /**
+     * @brief Stops worker threads
+     */
+    void stopJobs();
+
+    /**
+     * @brief Cleanup allocated resources
+     */
+    void destroy();
+
+    /**
+     * @brief Accumulate detailed CU statistics from worker thread local data.
+     */
+    void collectStats();
+};
+
+// Utils
+
+/**
+ * @brief A workaround to init CTUs before processRowEncoder does the same,
+ * since the CUData is needed before the FrameEncoder initializes it
+ */
+void initCTU(CUData& ctu, int row, int col, CTUTask& task);
+
+};
+    
+#endif

x265-4.1.tar/source/encoder/weightPrediction.cpp -> x265-4.2.tar/source/encoder/weightPrediction.cpp Changed

x265-4.1.tar/source/input/input.h -> x265-4.2.tar/source/input/input.h Changed

x265-4.1.tar/source/input/y4m.cpp -> x265-4.2.tar/source/input/y4m.cpp Changed

x265-4.1.tar/source/input/yuv.cpp -> x265-4.2.tar/source/input/yuv.cpp Changed

@@ -30,6 +30,7 @@
 #define ENABLE_THREADING 1
 
 #if _WIN32
+#define strncasecmp _strnicmp
 #include <io.h>
 #include <fcntl.h>
 #if defined(_MSC_VER)
@@ -53,6 +54,11 @@
     threadActive = false;
     ifs = NULL;
 
+    if (colorSpace < 0 || colorSpace >= X265_CSP_MAX)
+    {
+        x265_log(NULL, X265_LOG_ERROR, "Invalid color space: %d\n", colorSpace);
+        return;
+    }
     uint32_t pixelbytes = depth > 8 ? 2 : 1;
     framesize = 0;
     for (int i = 0; i < x265_cli_cspscolorSpace.planes + alphaAvailable; i++)
@@ -99,7 +105,11 @@
 
     info.frameCount = -1;
     /* try to estimate frame count, if this is not stdin */
+#if _WIN32
+    if (ifs != stdin && strncasecmp(info.filename, "\\\\.\\pipe\\", 9))
+#else
     if (ifs != stdin)
+#endif
     {
         int64_t cur = ftello(ifs);
         if (cur >= 0)
@@ -113,7 +123,11 @@
     }
     if (info.skipFrames)
     {
+#if _WIN32
+        if (ifs != stdin && strncasecmp(info.filename, "\\\\.\\pipe\\", 9))
+#else
         if (ifs != stdin)
+#endif
             fseeko(ifs, (int64_t)framesize * info.skipFrames, SEEK_CUR);
         else
             for (int i = 0; i < info.skipFrames; i++)

x265-4.1.tar/source/output/reconplay.cpp -> x265-4.2.tar/source/output/reconplay.cpp Changed

x265-4.1.tar/source/output/yuv.cpp -> x265-4.2.tar/source/output/yuv.cpp Changed

x265-4.1.tar/source/x265.cpp -> x265-4.2.tar/source/x265.cpp Changed

x265-4.1.tar/source/x265.h -> x265-4.2.tar/source/x265.h Changed

@@ -276,6 +276,8 @@
     double           decideWaitTime;
     double           row0WaitTime;
     double           wallTime;
+    int64_t          tmeTime;
+    int64_t          tmeWaitTime;
     double           refWaitWallTime;
     double           totalCTUTime;
     double           stallTime;
@@ -555,9 +557,14 @@
 #define X265_CPU_NEON            (1 << 1)   /* ARM NEON */
 #define X265_CPU_FAST_NEON_MRC   (1 << 2)   /* Transfer from NEON to ARM register is fast (Cortex-A9) */
 #define X265_CPU_SVE2            (1 << 3)   /* AArch64 SVE2 */
-#define X265_CPU_SVE             (1 << 4)   /* AArch64 SVE2 */
+#define X265_CPU_SVE             (1 << 4)   /* AArch64 SVE */
 #define X265_CPU_NEON_DOTPROD    (1 << 5)   /* AArch64 Neon DotProd */
 #define X265_CPU_NEON_I8MM       (1 << 6)   /* AArch64 Neon I8MM */
+#define X265_CPU_SVE2_BITPERM    (1 << 7)   /* AArch64 SVE2 BitPerm */
+
+/* RISCV */
+#define X265_CPU_RVV             (1 << 0)   /* RISCV vector */
+#define X265_CPU_ZBB             (1 << 1)   /* RISCV zbb */
 
 /* IBM Power8 */
 #define X265_CPU_ALTIVEC         0x0000001
@@ -746,7 +753,7 @@
                                                     "log316", "iec61966-2-4", "bt1361e", "iec61966-2-1", "bt2020-10", "bt2020-12",
                                                     "smpte2084", "smpte428", "arib-std-b67", 0 };
 static const char * const x265_colmatrix_names = { "gbr", "bt709", "unknown", "", "fcc", "bt470bg", "smpte170m", "smpte240m",
-                                                     "ycgco", "bt2020nc", "bt2020c", "smpte2085", "chroma-derived-nc", "chroma-derived-c", "ictcp", 0 };
+                                                     "ycgco", "bt2020nc", "bt2020c", "smpte2085", "chroma-derived-nc", "chroma-derived-c", "ictcp", "ipt-pq-c2", 0 };
 static const char * const x265_sar_names = { "unknown", "1:1", "12:11", "10:11", "16:11", "40:33", "24:11", "20:11",
                                                "32:11", "80:33", "18:11", "15:11", "64:33", "160:99", "4:3", "3:2", "2:1", 0 };
 static const char * const x265_interlace_names = { "prog", "tff", "bff", 0 };
@@ -1167,6 +1174,15 @@
      * win, particularly in video sequences with low motion. Default disabled */
     int       bDistributeMotionEstimation;
 
+    /* Use a dedicated threadpool to pre-process motion estimation. Evaluates all
+     * PU combinations for CTUs in parallel. Dependencies between CTUs in inter
+     * frames is broken to allow for more parallelism, and as result may cause
+     * drop in compression efficiency. Recommended for many core CPUs and when
+     * loss in compression efficiency is acceptable for speedup of encoding.
+     * Default disabled.
+     */
+    int       bThreadedME;
+
     /*== Logging Features ==*/
 
     /* Enable analysis and logging distribution of CUs. Now deprecated */
@@ -2324,6 +2340,13 @@
     int      searchRangeForLayer1;
     int      searchRangeForLayer2;
 
+    /* Threaded ME */
+    /* Number of CTUs processed at once when a worker thread picks up a task from ThreadedME. */
+    int      tmeTaskBlockSize;
+    
+    /* Number of rows upto which ThreadedME processes tasks ahead of WPP */
+    int      tmeNumBufferRows;
+
     /*SBRC*/
     int      bEnableSBRC;
     int mcstfFrameRange;
@@ -2344,6 +2367,9 @@
     /*Frame level RateControl Configuration*/
     int     bConfigRCFrame;
     int    isAbrLadderEnable;
+
+    /*tune*/
+    const char* tune;
 } x265_param;
 
 /* x265_param_alloc:

x265-4.1.tar/source/x265_config.h.in -> x265-4.2.tar/source/x265_config.h.in Changed

x265-4.1.tar/source/x265cli.cpp -> x265-4.2.tar/source/x265cli.cpp Changed

@@ -103,8 +103,9 @@
         H0("-F/--frame-threads <integer>     Number of concurrently encoded frames. 0: auto-determined by core count\n");
         H0("   --no-wpp                    Enable Wavefront Parallel Processing. Default %s\n", OPT(param->bEnableWavefront));
         H0("   --no-slices <integer>       Enable Multiple Slices feature. Default %d\n", param->maxSlices);
-        H0("   --no-pmode                  Parallel mode analysis. Default %s\n", OPT(param->bDistributeModeAnalysis));
-        H0("   --no-pme                    Parallel motion estimation. Default %s\n", OPT(param->bDistributeMotionEstimation));
+        H0("   --no-pmode                  Parallel mode analysis. Deprecated from release 4.1. Default %s\n", OPT(param->bDistributeModeAnalysis));
+        H0("   --no-pme                    Parallel motion estimation. Deprecated from release 4.1. Default %s\n", OPT(param->bDistributeMotionEstimation));
+        H0("   --no-threaded-me            Enables standalone multi-threaded module for motion estimation at CTU level. Default %s\n", OPT(param->bThreadedME));
         H0("   --no-asm <bool|int|string>  Override CPU detection. Default: auto\n");
         H0("\nPresets:\n");
         H0("-p/--preset <string>             Trade off performance for compression efficiency. Default medium\n");
@@ -327,7 +328,7 @@
         H0("                                 smpte240m, linear, log100, log316, iec61966-2-4, bt1361e, iec61966-2-1,\n");
         H0("                                 bt2020-10, bt2020-12, smpte2084, smpte428, arib-std-b67. Default unknown\n");
         H1("   --colormatrix <string>        Specify color matrix setting from unknown, bt709, fcc, bt470bg, smpte170m,\n");
-        H1("                                 smpte240m, gbr, ycgco, bt2020nc, bt2020c, smpte2085, chroma-derived-nc, chroma-derived-c, ictcp. Default unknown\n");
+        H1("                                 smpte240m, gbr, ycgco, bt2020nc, bt2020c, smpte2085, chroma-derived-nc, chroma-derived-c, ictcp, ipt-pq-c2. Default unknown\n");
         H1("   --chromaloc <integer>         Specify chroma sample location (0 to 5). Default of %d\n", param->vui.chromaSampleLocTypeTopField);
         H0("   --master-display <string>     SMPTE ST 2086 master display color volume info SEI (HDR)\n");
         H0("                                    format: G(x,y)B(x,y)R(x,y)WP(x,y)L(max,min)\n");
@@ -480,7 +481,21 @@
         if (output)
             output->release();
         output = NULL;
-    }
+        if (param && api)
+        {
+            api->param_free(param);
+            param = NULL;
+        }
+        // Free dynamically allocated input filenames
+        for (int i = 0; i < MAX_VIEWS; i++)
+        {
+            if (inputfni)
+            {
+                X265_FREE(inputfni);
+                inputfni = NULL;
+            }
+        }
+   }
 
     void CLIOptions::printStatus(uint32_t frameNum)
     {
@@ -617,12 +632,17 @@
 
     bool CLIOptions::parse(int argc, char **argv)
     {
+        if (argc <= 1)
+        {
+            x265_log(NULL, X265_LOG_ERROR, "No input file. Run x265 --help for a list of options.\n");
+            return true;
+        }
+
         bool bError = false;
         int bShowHelp = false;
         int inputBitDepth = 8;
         int outputBitDepth = 0;
         int reconFileBitDepth = 0;
-        char* inputfnMAX_VIEWS = { NULL };
         for (int view = 0; view < MAX_VIEWS; view++)
         {
             inputfnview = X265_MALLOC(char, sizeof(char) * 1024);
@@ -636,12 +656,6 @@
         int svtEnabled = 0;
         argCnt = argc;
 
-        if (argc <= 1)
-        {
-            x265_log(NULL, X265_LOG_ERROR, "No input file. Run x265 --help for a list of options.\n");
-            return true;
-        }
-
         /* Presets are applied before all other options. */
         for (optind = 0;;)
         {

x265-4.1.tar/source/x265cli.h -> x265-4.2.tar/source/x265cli.h Changed

x265-4.1.tar/x265Version.txt -> x265-4.2.tar/x265Version.txt Changed

Build Results
RPM Lint

Refresh

Comments for request 6522 0

Request History

Aloysius created request 4 days ago

Update to version 4.2

Aloysius accepted request 4 days ago