| Message ID | cover.1638865027.git.ming.qian@nxp.com |
|---|---|
| Headers | show |
| Series | amphion video decoder/encoder driver | expand |
Hi Ming, Running 'scripts/checkpatch.pl --strict' over the patches gives me lots of issues, most of them easily fixed. As always with checkpatch, use common sense since not everything can be fixed. You probably ran checkpatch.pl without the --strict option, but in the media subsystem we use that option. Note that sparse and smatch are happy, so it is just the checkpatch issues that need to be improved. On 12/7/21 09:24, Ming Qian wrote: > Hi all, > > This patch series adds support for > the amphion video encoder and decoder > via the VPU block present in imx8q platforms. > Currently, support for IMX8QXP and IMX8QM is included. > > It features decoding for the following formats: > - H.264 > - HEVC > - MPEG4 > - MPEG2 > - VC1 > - VP8 > > It features encoding for the following formats: > - H.264 > > The driver creates a separate device node for the encoder and decoder. > > This driver is dependent on vpu firmwares. > The firmwares have been submitted to linux-firmware. > The firmware patch is since commit > b563148fd28623f6b6ce68bb06c3dd3bd138b058: > linux-firmware: Update firmware file for Intel Bluetooth 9462 > (Fri Oct 8 16:30:14 2021 +0530) > > and it's available in the git repository at: > https://github.com/mingqian-0/linux-firmware.git > > for you to fetch changes up to bb3eee4f99589d4910dee4c053a3a685546b5dbb: > amphion: add VPU firmwares for NXP i.MX8Q SoCs > (Tue Oct 12 15:09:57 2021 +0800) > > encoder is tested with gstreamer > decoder is tested with gstreamer, but the following patches are required: > https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1379 > https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 > > > Tested-by: Nicolas Dufresne <nicolas.dufresne@collabora.com> For v15 please add this tag from Nicolas to the patch(es) that it applies to. Tags in the cover letter are ignored. Regarding patch 12/13 (firmware patch): who will take that? I think it should go through Shawn Guo, or if he can Ack it, then I can take it with the rest of the series as well. Regards, Hans > > > Changelog: > > v14 > - fix some errors according to Hans's comments > > v13 > - make a workaround that avoid firmware entering wfi wrongly > > v12 > - support reset decoder when starting a new stream > - don't append an empty last buffer, set last_buffer_dequeued > - improve the resolution change flow > - return all buffers if start_streaming fail > - fill encoder capture buffer's filed to none > - fix a bug in calculating bytesperline > > v11 > - fix dt_binding_check error after upgrade dtschema > - remove "default y" > - add media device > > v10 > - refine vpu log, remove custom logging infrastructure > - support non contiguous planes format nv12m instead of nv12 > - rename V4L2_PIX_FMT_NV12_8L128 to V4L2_PIX_FMT_NV12MT_8L128 > - rename V4L2_PIX_FMT_NV12_10BE_8L128 to V4L2_PIX_FMT_NV12MT_10BE_8L128 > - merge two module into one > - fix kernel panic in rmmod > > v9 > - drop V4L2_BUF_FLAG_CODECCONFIG > - drop V4L2_EVENT_CODEC_ERROR > - drop V4L2_EVENT_SKIP - use the v4l2_buffer.sequence counter > - fix some build warnings with W=1 reported by kernel test robot > > v8 > - move driver from driver/media/platform/imx/vpu-8q to > driver/media/platform/amphion > - rename driver name to amphion > - remove imx_vpu.h > - move the definition of V4L2_EVENT_CODEC_ERROR to videodev2.h > - move the definition of V4L2_EVENT_SKIP to videodev2.h > > v7 > - fix build warnings with W=1 reported by kernel test robot > > v6: > - rename V4L2_PIX_FMT_NT8 to V4L2_PIX_FMT_NV12_8L128 > - rename V4L2_PIX_FMT_NT10 to V4L2_PIX_FMT_NV12_10BE_8L128 > > v5: > - move some definition from imx_vph.h to videodev2.h > - remove some unnecessary content > - add some documentation descriptions > - pass the lateset v4l2-compliance test > > v4: > - redefine the memory-region in devicetree bindings documentation > - use v4l2's mechanism to implement synchronize queuing ioctl > - remove the unnecessary mutex ioctl_sync > - don't notify source change event if the parameters are same as previously established > - add flag V4L2_FMT_FLAG_DYN_RESOLUTION to decoder's capture format > > v3: > - don't make vpu device node a simple-bus > - trigger probing vpu core in the driver > - remove unnecessary vpu core index property > > v2: > - fix dt bindings build error > - split driver patch into several parts to avoid exceeding bytes limit > > Compliance > ========== > # v4l2-compliance -d /dev/video0 > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t > v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 > > Compliance test for amphion-vpu device /dev/video0: > > Driver Info: > Driver name : amphion-vpu > Card type : amphion vpu decoder > Bus info : platform: amphion-vpu > Driver version : 5.15.0 > Capabilities : 0x84204000 > Video Memory-to-Memory Multiplanar > Streaming > Extended Pix Format > Device Capabilities > Device Caps : 0x04204000 > Video Memory-to-Memory Multiplanar > Streaming > Extended Pix Format > Detected Stateful Decoder > Media Driver Info: > Driver name : amphion-vpu > Model : amphion-vpu > Serial : > Bus info : platform: amphion-vpu > Media version : 5.15.0 > Hardware revision: 0x00000000 (0) > Driver version : 5.15.0 > Interface Info: > ID : 0x0300000c > Type : V4L Video > Entity Info: > ID : 0x00000001 (1) > Name : amphion-vpu-decoder-source > Function : V4L2 I/O > Pad 0x01000002 : 0: Source > Link 0x02000008: to remote pad 0x1000004 of entity 'amphion-vpu-decoder-proc' (Video Decoder): Data, Enabled, Immutable > > Required ioctls: > test MC information (see 'Media Driver Info' above): OK > test VIDIOC_QUERYCAP: OK > test invalid ioctls: OK > > Allow for multiple opens: > test second /dev/video0 open: OK > test VIDIOC_QUERYCAP: OK > test VIDIOC_G/S_PRIORITY: OK > test for unlimited opens: OK > > Debug ioctls: > test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) > test VIDIOC_LOG_STATUS: OK (Not Supported) > > Input ioctls: > test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) > test VIDIOC_ENUMAUDIO: OK (Not Supported) > test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) > test VIDIOC_G/S_AUDIO: OK (Not Supported) > Inputs: 0 Audio Inputs: 0 Tuners: 0 > > Output ioctls: > test VIDIOC_G/S_MODULATOR: OK (Not Supported) > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > test VIDIOC_ENUMAUDOUT: OK (Not Supported) > test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) > test VIDIOC_G/S_AUDOUT: OK (Not Supported) > Outputs: 0 Audio Outputs: 0 Modulators: 0 > > Input/Output configuration ioctls: > test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) > test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) > test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) > test VIDIOC_G/S_EDID: OK (Not Supported) > > Control ioctls: > test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK > test VIDIOC_QUERYCTRL: OK > test VIDIOC_G/S_CTRL: OK > test VIDIOC_G/S/TRY_EXT_CTRLS: OK > test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK > test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) > Standard Controls: 3 Private Controls: 0 > > Format ioctls: > test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK > test VIDIOC_G/S_PARM: OK (Not Supported) > test VIDIOC_G_FBUF: OK (Not Supported) > test VIDIOC_G_FMT: OK > test VIDIOC_TRY_FMT: OK > test VIDIOC_S_FMT: OK > test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) > test Cropping: OK (Not Supported) > test Composing: OK > test Scaling: OK (Not Supported) > > Codec ioctls: > test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) > test VIDIOC_G_ENC_INDEX: OK (Not Supported) > test VIDIOC_(TRY_)DECODER_CMD: OK > > Buffer ioctls: > test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK > test VIDIOC_EXPBUF: OK > test Requests: OK (Not Supported) > > Total for amphion-vpu device /dev/video0: 46, Succeeded: 46, Failed: 0, Warnings: 0 > > # v4l2-compliance -d /dev/video1 > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t > v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 > > Compliance test for amphion-vpu device /dev/video1: > > Driver Info: > Driver name : amphion-vpu > Card type : amphion vpu encoder > Bus info : platform: amphion-vpu > Driver version : 5.15.0 > Capabilities : 0x84204000 > Video Memory-to-Memory Multiplanar > Streaming > Extended Pix Format > Device Capabilities > Device Caps : 0x04204000 > Video Memory-to-Memory Multiplanar > Streaming > Extended Pix Format > Detected Stateful Encoder > Media Driver Info: > Driver name : amphion-vpu > Model : amphion-vpu > Serial : > Bus info : platform: amphion-vpu > Media version : 5.15.0 > Hardware revision: 0x00000000 (0) > Driver version : 5.15.0 > Interface Info: > ID : 0x0300001a > Type : V4L Video > Entity Info: > ID : 0x0000000f (15) > Name : amphion-vpu-encoder-source > Function : V4L2 I/O > Pad 0x01000010 : 0: Source > Link 0x02000016: to remote pad 0x1000012 of entity 'amphion-vpu-encoder-proc' (Video Encoder): Data, Enabled, Immutable > > Required ioctls: > test MC information (see 'Media Driver Info' above): OK > test VIDIOC_QUERYCAP: OK > test invalid ioctls: OK > > Allow for multiple opens: > test second /dev/video1 open: OK > test VIDIOC_QUERYCAP: OK > test VIDIOC_G/S_PRIORITY: OK > test for unlimited opens: OK > > Debug ioctls: > test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) > test VIDIOC_LOG_STATUS: OK (Not Supported) > > Input ioctls: > test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) > test VIDIOC_ENUMAUDIO: OK (Not Supported) > test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) > test VIDIOC_G/S_AUDIO: OK (Not Supported) > Inputs: 0 Audio Inputs: 0 Tuners: 0 > > Output ioctls: > test VIDIOC_G/S_MODULATOR: OK (Not Supported) > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > test VIDIOC_ENUMAUDOUT: OK (Not Supported) > test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) > test VIDIOC_G/S_AUDOUT: OK (Not Supported) > Outputs: 0 Audio Outputs: 0 Modulators: 0 > > Input/Output configuration ioctls: > test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) > test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) > test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) > test VIDIOC_G/S_EDID: OK (Not Supported) > > Control ioctls: > test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK > test VIDIOC_QUERYCTRL: OK > test VIDIOC_G/S_CTRL: OK > test VIDIOC_G/S/TRY_EXT_CTRLS: OK > test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK > test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) > Standard Controls: 20 Private Controls: 0 > > Format ioctls: > test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK > test VIDIOC_G/S_PARM: OK > test VIDIOC_G_FBUF: OK (Not Supported) > test VIDIOC_G_FMT: OK > test VIDIOC_TRY_FMT: OK > test VIDIOC_S_FMT: OK > test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) > test Cropping: OK > test Composing: OK (Not Supported) > test Scaling: OK (Not Supported) > > Codec ioctls: > test VIDIOC_(TRY_)ENCODER_CMD: OK > test VIDIOC_G_ENC_INDEX: OK (Not Supported) > test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported) > > Buffer ioctls: > test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK > test VIDIOC_EXPBUF: OK > test Requests: OK (Not Supported) > > Total for amphion-vpu device /dev/video1: 46, Succeeded: 46, Failed: 0, Warnings: 0 > > # v4l2-compliance -d /dev/media0 > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t > v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 > > Compliance test for amphion-vpu device /dev/media0: > > Media Driver Info: > Driver name : amphion-vpu > Model : amphion-vpu > Serial : > Bus info : platform: amphion-vpu > Media version : 5.15.0 > Hardware revision: 0x00000000 (0) > Driver version : 5.15.0 > > Required ioctls: > test MEDIA_IOC_DEVICE_INFO: OK > test invalid ioctls: OK > > Allow for multiple opens: > test second /dev/media0 open: OK > test MEDIA_IOC_DEVICE_INFO: OK > test for unlimited opens: OK > > Media Controller ioctls: > test MEDIA_IOC_G_TOPOLOGY: OK > Entities: 6 Interfaces: 2 Pads: 8 Links: 8 > test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK > test MEDIA_IOC_SETUP_LINK: OK > > Total for amphion-vpu device /dev/media0: 8, Succeeded: 8, Failed: 0, Warnings: 0 > > Ming Qian (13): > dt-bindings: media: amphion: add amphion video codec bindings > media: add nv12m_8l128 and nv12m_10be_8l128 video format. > media: amphion: add amphion vpu device driver > media: amphion: add vpu core driver > media: amphion: implement vpu core communication based on mailbox > media: amphion: add vpu v4l2 m2m support > media: amphion: add v4l2 m2m vpu encoder stateful driver > media: amphion: add v4l2 m2m vpu decoder stateful driver > media: amphion: implement windsor encoder rpc interface > media: amphion: implement malone decoder rpc interface > ARM64: dts: freescale: imx8q: add imx vpu codec entries > firmware: imx: scu-pd: imx8q: add vpu mu resources > MAINTAINERS: add AMPHION VPU CODEC V4L2 driver entry > > .../bindings/media/amphion,vpu.yaml | 180 ++ > .../media/v4l/pixfmt-yuv-planar.rst | 28 +- > MAINTAINERS | 9 + > .../arm64/boot/dts/freescale/imx8-ss-vpu.dtsi | 72 + > arch/arm64/boot/dts/freescale/imx8qxp-mek.dts | 17 + > arch/arm64/boot/dts/freescale/imx8qxp.dtsi | 24 + > arch/arm64/configs/defconfig | 1 + > drivers/firmware/imx/scu-pd.c | 4 + > drivers/media/platform/Kconfig | 19 + > drivers/media/platform/Makefile | 2 + > drivers/media/platform/amphion/Makefile | 20 + > drivers/media/platform/amphion/vdec.c | 1692 +++++++++++++++++ > drivers/media/platform/amphion/venc.c | 1354 +++++++++++++ > drivers/media/platform/amphion/vpu.h | 356 ++++ > drivers/media/platform/amphion/vpu_cmds.c | 437 +++++ > drivers/media/platform/amphion/vpu_cmds.h | 25 + > drivers/media/platform/amphion/vpu_codec.h | 67 + > drivers/media/platform/amphion/vpu_color.c | 186 ++ > drivers/media/platform/amphion/vpu_core.c | 872 +++++++++ > drivers/media/platform/amphion/vpu_core.h | 15 + > drivers/media/platform/amphion/vpu_dbg.c | 497 +++++ > drivers/media/platform/amphion/vpu_defs.h | 186 ++ > drivers/media/platform/amphion/vpu_drv.c | 260 +++ > drivers/media/platform/amphion/vpu_helpers.c | 416 ++++ > drivers/media/platform/amphion/vpu_helpers.h | 70 + > drivers/media/platform/amphion/vpu_imx8q.c | 271 +++ > drivers/media/platform/amphion/vpu_imx8q.h | 116 ++ > drivers/media/platform/amphion/vpu_malone.c | 1630 ++++++++++++++++ > drivers/media/platform/amphion/vpu_malone.h | 42 + > drivers/media/platform/amphion/vpu_mbox.c | 118 ++ > drivers/media/platform/amphion/vpu_mbox.h | 16 + > drivers/media/platform/amphion/vpu_msgs.c | 388 ++++ > drivers/media/platform/amphion/vpu_msgs.h | 14 + > drivers/media/platform/amphion/vpu_rpc.c | 265 +++ > drivers/media/platform/amphion/vpu_rpc.h | 453 +++++ > drivers/media/platform/amphion/vpu_v4l2.c | 733 +++++++ > drivers/media/platform/amphion/vpu_v4l2.h | 56 + > drivers/media/platform/amphion/vpu_windsor.c | 1207 ++++++++++++ > drivers/media/platform/amphion/vpu_windsor.h | 39 + > drivers/media/v4l2-core/v4l2-ioctl.c | 2 + > include/uapi/linux/videodev2.h | 2 + > 41 files changed, 12158 insertions(+), 3 deletions(-) > create mode 100644 Documentation/devicetree/bindings/media/amphion,vpu.yaml > create mode 100644 arch/arm64/boot/dts/freescale/imx8-ss-vpu.dtsi > create mode 100644 drivers/media/platform/amphion/Makefile > create mode 100644 drivers/media/platform/amphion/vdec.c > create mode 100644 drivers/media/platform/amphion/venc.c > create mode 100644 drivers/media/platform/amphion/vpu.h > create mode 100644 drivers/media/platform/amphion/vpu_cmds.c > create mode 100644 drivers/media/platform/amphion/vpu_cmds.h > create mode 100644 drivers/media/platform/amphion/vpu_codec.h > create mode 100644 drivers/media/platform/amphion/vpu_color.c > create mode 100644 drivers/media/platform/amphion/vpu_core.c > create mode 100644 drivers/media/platform/amphion/vpu_core.h > create mode 100644 drivers/media/platform/amphion/vpu_dbg.c > create mode 100644 drivers/media/platform/amphion/vpu_defs.h > create mode 100644 drivers/media/platform/amphion/vpu_drv.c > create mode 100644 drivers/media/platform/amphion/vpu_helpers.c > create mode 100644 drivers/media/platform/amphion/vpu_helpers.h > create mode 100644 drivers/media/platform/amphion/vpu_imx8q.c > create mode 100644 drivers/media/platform/amphion/vpu_imx8q.h > create mode 100644 drivers/media/platform/amphion/vpu_malone.c > create mode 100644 drivers/media/platform/amphion/vpu_malone.h > create mode 100644 drivers/media/platform/amphion/vpu_mbox.c > create mode 100644 drivers/media/platform/amphion/vpu_mbox.h > create mode 100644 drivers/media/platform/amphion/vpu_msgs.c > create mode 100644 drivers/media/platform/amphion/vpu_msgs.h > create mode 100644 drivers/media/platform/amphion/vpu_rpc.c > create mode 100644 drivers/media/platform/amphion/vpu_rpc.h > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.c > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.h > create mode 100644 drivers/media/platform/amphion/vpu_windsor.c > create mode 100644 drivers/media/platform/amphion/vpu_windsor.h > > > base-commit: 1f1517fafda598839a02e39968c5063ddcfa51fc
Hi Ming, On 12/7/21 09:24, Ming Qian wrote: > vpu_v4l2.c implements the v4l2 m2m driver methods. > vpu_helpers.c implements the common helper functions > vpu_color.c converts the v4l2 colorspace with > the VUI parameters that specified by ITU-T | ISO/IEC > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > Signed-off-by: Shijie Qin <shijie.qin@nxp.com> > Signed-off-by: Zhou Peng <eagle.zhou@nxp.com> > Reported-by: kernel test robot <lkp@intel.com> > --- > drivers/media/platform/amphion/vpu_color.c | 186 +++++ > drivers/media/platform/amphion/vpu_helpers.c | 416 +++++++++++ > drivers/media/platform/amphion/vpu_helpers.h | 70 ++ > drivers/media/platform/amphion/vpu_v4l2.c | 733 +++++++++++++++++++ > drivers/media/platform/amphion/vpu_v4l2.h | 56 ++ > 5 files changed, 1461 insertions(+) > create mode 100644 drivers/media/platform/amphion/vpu_color.c > create mode 100644 drivers/media/platform/amphion/vpu_helpers.c > create mode 100644 drivers/media/platform/amphion/vpu_helpers.h > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.c > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.h > > diff --git a/drivers/media/platform/amphion/vpu_color.c b/drivers/media/platform/amphion/vpu_color.c > new file mode 100644 > index 000000000000..1c062dc40306 > --- /dev/null > +++ b/drivers/media/platform/amphion/vpu_color.c > @@ -0,0 +1,186 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#include <linux/init.h> > +#include <linux/device.h> > +#include <linux/ioctl.h> > +#include <linux/list.h> > +#include <linux/module.h> > +#include <linux/kernel.h> > +#include <linux/slab.h> > +#include <linux/delay.h> > +#include <linux/types.h> > +#include <media/v4l2-device.h> > +#include "vpu.h" > +#include "vpu_helpers.h" > + > +static const u8 colorprimaries[] = { > + 0, > + V4L2_COLORSPACE_REC709, /*Rec. ITU-R BT.709-6*/ > + 0, > + 0, > + V4L2_COLORSPACE_470_SYSTEM_M, /*Rec. ITU-R BT.470-6 System M*/ > + V4L2_COLORSPACE_470_SYSTEM_BG, /*Rec. ITU-R BT.470-6 System B, G*/ > + V4L2_COLORSPACE_SMPTE170M, /*SMPTE170M*/ > + V4L2_COLORSPACE_SMPTE240M, /*SMPTE240M*/ > + 0, /*Generic film*/ > + V4L2_COLORSPACE_BT2020, /*Rec. ITU-R BT.2020-2*/ > + 0, /*SMPTE ST 428-1*/ > +}; > + > +static const u8 colortransfers[] = { > + 0, > + V4L2_XFER_FUNC_709, /*Rec. ITU-R BT.709-6*/ > + 0, > + 0, > + 0, /*Rec. ITU-R BT.470-6 System M*/ > + 0, /*Rec. ITU-R BT.470-6 System B, G*/ > + V4L2_XFER_FUNC_709, /*SMPTE170M*/ > + V4L2_XFER_FUNC_SMPTE240M, /*SMPTE240M*/ > + V4L2_XFER_FUNC_NONE, /*Linear transfer characteristics*/ > + 0, > + 0, > + 0, /*IEC 61966-2-4*/ > + 0, /*Rec. ITU-R BT.1361-0 extended colour gamut*/ > + V4L2_XFER_FUNC_SRGB, /*IEC 61966-2-1 sRGB or sYCC*/ > + V4L2_XFER_FUNC_709, /*Rec. ITU-R BT.2020-2 (10 bit system)*/ > + V4L2_XFER_FUNC_709, /*Rec. ITU-R BT.2020-2 (12 bit system)*/ > + V4L2_XFER_FUNC_SMPTE2084, /*SMPTE ST 2084*/ > + 0, /*SMPTE ST 428-1*/ > + 0 /*Rec. ITU-R BT.2100-0 hybrid log-gamma (HLG)*/ > +}; > + > +static const u8 colormatrixcoefs[] = { > + 0, > + V4L2_YCBCR_ENC_709, /*Rec. ITU-R BT.709-6*/ > + 0, > + 0, > + 0, /*Title 47 Code of Federal Regulations*/ > + V4L2_YCBCR_ENC_601, /*Rec. ITU-R BT.601-7 625*/ > + V4L2_YCBCR_ENC_601, /*Rec. ITU-R BT.601-7 525*/ > + V4L2_YCBCR_ENC_SMPTE240M, /*SMPTE240M*/ > + 0, > + V4L2_YCBCR_ENC_BT2020, /*Rec. ITU-R BT.2020-2*/ > + V4L2_YCBCR_ENC_BT2020_CONST_LUM /*Rec. ITU-R BT.2020-2 constant*/ > +}; > + > +u32 vpu_color_cvrt_primaries_v2i(u32 primaries) > +{ > + return VPU_ARRAY_FIND(colorprimaries, primaries); > +} > + > +u32 vpu_color_cvrt_primaries_i2v(u32 primaries) > +{ > + return VPU_ARRAY_AT(colorprimaries, primaries); > +} > + > +u32 vpu_color_cvrt_transfers_v2i(u32 transfers) > +{ > + return VPU_ARRAY_FIND(colortransfers, transfers); > +} > + > +u32 vpu_color_cvrt_transfers_i2v(u32 transfers) > +{ > + return VPU_ARRAY_AT(colortransfers, transfers); > +} > + > +u32 vpu_color_cvrt_matrix_v2i(u32 matrix) > +{ > + return VPU_ARRAY_FIND(colormatrixcoefs, matrix); > +} > + > +u32 vpu_color_cvrt_matrix_i2v(u32 matrix) > +{ > + return VPU_ARRAY_AT(colormatrixcoefs, matrix); > +} > + > +u32 vpu_color_cvrt_full_range_v2i(u32 full_range) > +{ > + return (full_range == V4L2_QUANTIZATION_FULL_RANGE); > +} > + > +u32 vpu_color_cvrt_full_range_i2v(u32 full_range) > +{ > + if (full_range) > + return V4L2_QUANTIZATION_FULL_RANGE; > + > + return V4L2_QUANTIZATION_LIM_RANGE; > +} > + > +int vpu_color_check_primaries(u32 primaries) > +{ > + return vpu_color_cvrt_primaries_v2i(primaries) ? 0 : -EINVAL; > +} > + > +int vpu_color_check_transfers(u32 transfers) > +{ > + return vpu_color_cvrt_transfers_v2i(transfers) ? 0 : -EINVAL; > +} > + > +int vpu_color_check_matrix(u32 matrix) > +{ > + return vpu_color_cvrt_matrix_v2i(matrix) ? 0 : -EINVAL; > +} > + > +int vpu_color_check_full_range(u32 full_range) > +{ > + int ret = -EINVAL; > + > + switch (full_range) { > + case V4L2_QUANTIZATION_FULL_RANGE: > + case V4L2_QUANTIZATION_LIM_RANGE: > + ret = 0; > + break; > + default: > + break; > + > + } > + > + return ret; > +} > + > +int vpu_color_get_default(u32 primaries, > + u32 *ptransfers, u32 *pmatrix, u32 *pfull_range) > +{ > + u32 transfers; > + u32 matrix; > + u32 full_range; > + > + switch (primaries) { > + case V4L2_COLORSPACE_REC709: > + transfers = V4L2_XFER_FUNC_709; > + matrix = V4L2_YCBCR_ENC_709; > + break; > + case V4L2_COLORSPACE_470_SYSTEM_M: > + case V4L2_COLORSPACE_470_SYSTEM_BG: > + case V4L2_COLORSPACE_SMPTE170M: > + transfers = V4L2_XFER_FUNC_709; > + matrix = V4L2_YCBCR_ENC_601; > + break; > + case V4L2_COLORSPACE_SMPTE240M: > + transfers = V4L2_XFER_FUNC_SMPTE240M; > + matrix = V4L2_YCBCR_ENC_SMPTE240M; > + break; > + case V4L2_COLORSPACE_BT2020: > + transfers = V4L2_XFER_FUNC_709; > + matrix = V4L2_YCBCR_ENC_BT2020; > + break; > + default: > + transfers = V4L2_XFER_FUNC_DEFAULT; > + matrix = V4L2_YCBCR_ENC_DEFAULT; > + break; > + } > + full_range = V4L2_QUANTIZATION_LIM_RANGE; > + > + if (ptransfers) > + *ptransfers = transfers; > + if (pmatrix) > + *pmatrix = matrix; > + if (pfull_range) > + *pfull_range = full_range; > + > + > + return 0; > +} > diff --git a/drivers/media/platform/amphion/vpu_helpers.c b/drivers/media/platform/amphion/vpu_helpers.c > new file mode 100644 > index 000000000000..f41df259eb1e > --- /dev/null > +++ b/drivers/media/platform/amphion/vpu_helpers.c > @@ -0,0 +1,416 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#include <linux/init.h> > +#include <linux/interconnect.h> > +#include <linux/ioctl.h> > +#include <linux/list.h> > +#include <linux/kernel.h> > +#include <linux/module.h> > +#include <linux/platform_device.h> > +#include "vpu.h" > +#include "vpu_core.h" > +#include "vpu_rpc.h" > +#include "vpu_helpers.h" > + > +int vpu_helper_find_in_array_u8(const u8 *array, u32 size, u32 x) > +{ > + int i; > + > + for (i = 0; i < size; i++) { > + if (array[i] == x) > + return i; > + } > + > + return 0; > +} > + > +bool vpu_helper_check_type(struct vpu_inst *inst, u32 type) > +{ > + const struct vpu_format *pfmt; > + > + for (pfmt = inst->formats; pfmt->pixfmt; pfmt++) { > + if (!vpu_iface_check_format(inst, pfmt->pixfmt)) > + continue; > + if (pfmt->type == type) > + return true; > + } > + > + return false; > +} > + > +const struct vpu_format *vpu_helper_find_format(struct vpu_inst *inst, u32 type, u32 pixelfmt) > +{ > + const struct vpu_format *pfmt; > + > + if (!inst || !inst->formats) > + return NULL; > + > + if (!vpu_iface_check_format(inst, pixelfmt)) > + return NULL; > + > + for (pfmt = inst->formats; pfmt->pixfmt; pfmt++) { > + if (pfmt->pixfmt == pixelfmt && (!type || type == pfmt->type)) > + return pfmt; > + } > + > + return NULL; > +} > + > +const struct vpu_format *vpu_helper_enum_format(struct vpu_inst *inst, u32 type, int index) > +{ > + const struct vpu_format *pfmt; > + int i = 0; > + > + if (!inst || !inst->formats) > + return NULL; > + > + for (pfmt = inst->formats; pfmt->pixfmt; pfmt++) { > + if (!vpu_iface_check_format(inst, pfmt->pixfmt)) > + continue; > + > + if (pfmt->type == type) { > + if (index == i) > + return pfmt; > + i++; > + } > + } > + > + return NULL; > +} > + > +u32 vpu_helper_valid_frame_width(struct vpu_inst *inst, u32 width) > +{ > + const struct vpu_core_resources *res; > + > + if (!inst) > + return width; > + > + res = vpu_get_resource(inst); > + if (!res) > + return width; > + if (res->max_width) > + width = clamp(width, res->min_width, res->max_width); > + if (res->step_width) > + width = ALIGN(width, res->step_width); > + > + return width; > +} > + > +u32 vpu_helper_valid_frame_height(struct vpu_inst *inst, u32 height) > +{ > + const struct vpu_core_resources *res; > + > + if (!inst) > + return height; > + > + res = vpu_get_resource(inst); > + if (!res) > + return height; > + if (res->max_height) > + height = clamp(height, res->min_height, res->max_height); > + if (res->step_height) > + height = ALIGN(height, res->step_height); > + > + return height; > +} > + > +static u32 get_nv12_plane_size(u32 width, u32 height, int plane_no, > + u32 stride, u32 interlaced, u32 *pbl) > +{ > + u32 bytesperline; > + u32 size = 0; > + > + bytesperline = ALIGN(width, stride); > + if (pbl) > + bytesperline = max(bytesperline, *pbl); > + height = ALIGN(height, 2); > + if (plane_no == 0) > + size = bytesperline * height; > + else if (plane_no == 1) > + size = bytesperline * height >> 1; > + if (pbl) > + *pbl = bytesperline; > + > + return size; > +} > + > +static u32 get_tiled_8l128_plane_size(u32 fmt, u32 width, u32 height, int plane_no, > + u32 stride, u32 interlaced, u32 *pbl) > +{ > + u32 ws = 3; > + u32 hs = 7; > + u32 bitdepth = 8; > + u32 bytesperline; > + u32 size = 0; > + > + if (interlaced) > + hs++; > + if (fmt == V4L2_PIX_FMT_NV12M_10BE_8L128) > + bitdepth = 10; > + bytesperline = DIV_ROUND_UP(width * bitdepth, BITS_PER_BYTE); > + bytesperline = ALIGN(bytesperline, 1 << ws); > + bytesperline = ALIGN(bytesperline, stride); > + if (pbl) > + bytesperline = max(bytesperline, *pbl); > + height = ALIGN(height, 1 << hs); > + if (plane_no == 0) > + size = bytesperline * height; > + else if (plane_no == 1) > + size = (bytesperline * ALIGN(height, 1 << (hs + 1))) >> 1; > + if (pbl) > + *pbl = bytesperline; > + > + return size; > +} > + > +static u32 get_default_plane_size(u32 width, u32 height, int plane_no, > + u32 stride, u32 interlaced, u32 *pbl) > +{ > + u32 bytesperline; > + u32 size = 0; > + > + bytesperline = ALIGN(width, stride); > + if (pbl) > + bytesperline = max(bytesperline, *pbl); > + if (plane_no == 0) > + size = bytesperline * height; > + if (pbl) > + *pbl = bytesperline; > + > + return size; > +} > + > +u32 vpu_helper_get_plane_size(u32 fmt, u32 w, u32 h, int plane_no, > + u32 stride, u32 interlaced, u32 *pbl) > +{ > + switch (fmt) { > + case V4L2_PIX_FMT_NV12M: > + return get_nv12_plane_size(w, h, plane_no, stride, interlaced, pbl); > + case V4L2_PIX_FMT_NV12M_8L128: > + case V4L2_PIX_FMT_NV12M_10BE_8L128: > + return get_tiled_8l128_plane_size(fmt, w, h, plane_no, stride, interlaced, pbl); > + default: > + return get_default_plane_size(w, h, plane_no, stride, interlaced, pbl); > + } > +} > + > +u32 vpu_helper_copy_from_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *rptr, u32 size, void *dst) > +{ > + u32 offset; > + u32 start; > + u32 end; > + void *virt; > + > + if (!stream_buffer || !rptr || !dst) > + return -EINVAL; > + > + if (!size) > + return 0; > + > + offset = *rptr; > + start = stream_buffer->phys; > + end = start + stream_buffer->length; > + virt = stream_buffer->virt; > + > + if (offset < start || offset > end) > + return -EINVAL; > + > + if (offset + size <= end) { > + memcpy(dst, virt + (offset - start), size); > + } else { > + memcpy(dst, virt + (offset - start), end - offset); > + memcpy(dst + end - offset, virt, size + offset - end); > + } > + > + *rptr = vpu_helper_step_walk(stream_buffer, offset, size); > + return size; > +} > + > +u32 vpu_helper_copy_to_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *wptr, u32 size, void *src) > +{ > + u32 offset; > + u32 start; > + u32 end; > + void *virt; > + > + if (!stream_buffer || !wptr || !src) > + return -EINVAL; > + > + if (!size) > + return 0; > + > + offset = *wptr; > + start = stream_buffer->phys; > + end = start + stream_buffer->length; > + virt = stream_buffer->virt; > + if (offset < start || offset > end) > + return -EINVAL; > + > + if (offset + size <= end) { > + memcpy(virt + (offset - start), src, size); > + } else { > + memcpy(virt + (offset - start), src, end - offset); > + memcpy(virt, src + end - offset, size + offset - end); > + } > + > + *wptr = vpu_helper_step_walk(stream_buffer, offset, size); > + > + return size; > +} > + > +u32 vpu_helper_memset_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *wptr, u8 val, u32 size) > +{ > + u32 offset; > + u32 start; > + u32 end; > + void *virt; > + > + if (!stream_buffer || !wptr) > + return -EINVAL; > + > + if (!size) > + return 0; > + > + offset = *wptr; > + start = stream_buffer->phys; > + end = start + stream_buffer->length; > + virt = stream_buffer->virt; > + if (offset < start || offset > end) > + return -EINVAL; > + > + if (offset + size <= end) { > + memset(virt + (offset - start), val, size); > + } else { > + memset(virt + (offset - start), val, end - offset); > + memset(virt, val, size + offset - end); > + } > + > + offset += size; > + if (offset >= end) > + offset -= stream_buffer->length; > + > + *wptr = offset; > + > + return size; > +} > + > +u32 vpu_helper_get_free_space(struct vpu_inst *inst) > +{ > + struct vpu_rpc_buffer_desc desc; > + > + if (vpu_iface_get_stream_buffer_desc(inst, &desc)) > + return 0; > + > + if (desc.rptr > desc.wptr) > + return desc.rptr - desc.wptr; > + else if (desc.rptr < desc.wptr) > + return (desc.end - desc.start + desc.rptr - desc.wptr); > + else > + return desc.end - desc.start; > +} > + > +u32 vpu_helper_get_used_space(struct vpu_inst *inst) > +{ > + struct vpu_rpc_buffer_desc desc; > + > + if (vpu_iface_get_stream_buffer_desc(inst, &desc)) > + return 0; > + > + if (desc.wptr > desc.rptr) > + return desc.wptr - desc.rptr; > + else if (desc.wptr < desc.rptr) > + return (desc.end - desc.start + desc.wptr - desc.rptr); > + else > + return 0; > +} > + > +int vpu_helper_g_volatile_ctrl(struct v4l2_ctrl *ctrl) > +{ > + struct vpu_inst *inst = ctrl_to_inst(ctrl); > + > + switch (ctrl->id) { > + case V4L2_CID_MIN_BUFFERS_FOR_CAPTURE: > + ctrl->val = inst->min_buffer_cap; > + break; > + case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT: > + ctrl->val = inst->min_buffer_out; > + break; > + default: > + return -EINVAL; > + } > + > + return 0; > +} > + > +#define READ_BYTE(buffer, pos) \ > + (*(u8 *)((buffer)->virt + ((pos) % buffer->length))) > + > +int vpu_helper_find_startcode(struct vpu_buffer *stream_buffer, > + u32 pixelformat, u32 offset, u32 bytesused) > +{ > + u32 start_code; > + int start_code_size; > + u32 val = 0; > + int i; > + int ret = -EINVAL; > + > + if (!stream_buffer || !stream_buffer->virt) > + return -EINVAL; > + > + switch (pixelformat) { > + case V4L2_PIX_FMT_H264: > + start_code_size = 4; > + start_code = 0x00000001; > + break; > + default: > + return 0; > + } > + > + for (i = 0; i < bytesused; i++) { > + val = (val << 8) | READ_BYTE(stream_buffer, offset + i); > + if (i < start_code_size - 1) > + continue; > + if (val == start_code) { > + ret = i + 1 - start_code_size; > + break; > + } > + } > + > + return ret; > +} > + > +int vpu_find_dst_by_src(struct vpu_pair *pairs, u32 cnt, u32 src) > +{ > + u32 i; > + > + if (!pairs || !cnt) > + return -EINVAL; > + > + for (i = 0; i < cnt; i++) { > + if (pairs[i].src == src) > + return pairs[i].dst; > + } > + > + return -EINVAL; > +} > + > +int vpu_find_src_by_dst(struct vpu_pair *pairs, u32 cnt, u32 dst) > +{ > + u32 i; > + > + if (!pairs || !cnt) > + return -EINVAL; > + > + for (i = 0; i < cnt; i++) { > + if (pairs[i].dst == dst) > + return pairs[i].src; > + } > + > + return -EINVAL; > +} > diff --git a/drivers/media/platform/amphion/vpu_helpers.h b/drivers/media/platform/amphion/vpu_helpers.h > new file mode 100644 > index 000000000000..8cb9b8c77745 > --- /dev/null > +++ b/drivers/media/platform/amphion/vpu_helpers.h > @@ -0,0 +1,70 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#ifndef _AMPHION_VPU_HELPERS_H > +#define _AMPHION_VPU_HELPERS_H > + > +struct vpu_pair { > + u32 src; > + u32 dst; > +}; > + > +#define MAKE_TIMESTAMP(s, ns) (((s32)(s) * NSEC_PER_SEC) + (ns)) > +#define VPU_INVALID_TIMESTAMP MAKE_TIMESTAMP(-1, 0) > +#define VPU_ARRAY_AT(array, i) (((i) < ARRAY_SIZE(array)) ? array[i] : 0) > +#define VPU_ARRAY_FIND(array, x) vpu_helper_find_in_array_u8(array, ARRAY_SIZE(array), x) > + > +int vpu_helper_find_in_array_u8(const u8 *array, u32 size, u32 x); > +bool vpu_helper_check_type(struct vpu_inst *inst, u32 type); > +const struct vpu_format *vpu_helper_find_format(struct vpu_inst *inst, u32 type, u32 pixelfmt); > +const struct vpu_format *vpu_helper_enum_format(struct vpu_inst *inst, u32 type, int index); > +u32 vpu_helper_valid_frame_width(struct vpu_inst *inst, u32 width); > +u32 vpu_helper_valid_frame_height(struct vpu_inst *inst, u32 height); > +u32 vpu_helper_get_plane_size(u32 fmt, u32 width, u32 height, int plane_no, > + u32 stride, u32 interlaced, u32 *pbl); > +u32 vpu_helper_copy_from_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *rptr, u32 size, void *dst); > +u32 vpu_helper_copy_to_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *wptr, u32 size, void *src); > +u32 vpu_helper_memset_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 *wptr, u8 val, u32 size); > +u32 vpu_helper_get_free_space(struct vpu_inst *inst); > +u32 vpu_helper_get_used_space(struct vpu_inst *inst); > +int vpu_helper_g_volatile_ctrl(struct v4l2_ctrl *ctrl); > +void vpu_helper_get_kmp_next(const u8 *pattern, int *next, int size); > +int vpu_helper_kmp_search(u8 *s, int s_len, const u8 *p, int p_len, int *next); > +int vpu_helper_kmp_search_in_stream_buffer(struct vpu_buffer *stream_buffer, > + u32 offset, int bytesused, > + const u8 *p, int p_len, int *next); > +int vpu_helper_find_startcode(struct vpu_buffer *stream_buffer, > + u32 pixelformat, u32 offset, u32 bytesused); > + > +static inline u32 vpu_helper_step_walk(struct vpu_buffer *stream_buffer, u32 pos, u32 step) > +{ > + pos += step; > + if (pos > stream_buffer->phys + stream_buffer->length) > + pos -= stream_buffer->length; > + > + return pos; > +} > + > +int vpu_color_check_primaries(u32 primaries); > +int vpu_color_check_transfers(u32 transfers); > +int vpu_color_check_matrix(u32 matrix); > +int vpu_color_check_full_range(u32 full_range); > +u32 vpu_color_cvrt_primaries_v2i(u32 primaries); > +u32 vpu_color_cvrt_primaries_i2v(u32 primaries); > +u32 vpu_color_cvrt_transfers_v2i(u32 transfers); > +u32 vpu_color_cvrt_transfers_i2v(u32 transfers); > +u32 vpu_color_cvrt_matrix_v2i(u32 matrix); > +u32 vpu_color_cvrt_matrix_i2v(u32 matrix); > +u32 vpu_color_cvrt_full_range_v2i(u32 full_range); > +u32 vpu_color_cvrt_full_range_i2v(u32 full_range); > +int vpu_color_get_default(u32 primaries, > + u32 *ptransfers, u32 *pmatrix, u32 *pfull_range); > + > +int vpu_find_dst_by_src(struct vpu_pair *pairs, u32 cnt, u32 src); > +int vpu_find_src_by_dst(struct vpu_pair *pairs, u32 cnt, u32 dst); > +#endif > diff --git a/drivers/media/platform/amphion/vpu_v4l2.c b/drivers/media/platform/amphion/vpu_v4l2.c > new file mode 100644 > index 000000000000..a91690f691bc > --- /dev/null > +++ b/drivers/media/platform/amphion/vpu_v4l2.c > @@ -0,0 +1,733 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#include <linux/init.h> > +#include <linux/interconnect.h> > +#include <linux/ioctl.h> > +#include <linux/list.h> > +#include <linux/kernel.h> > +#include <linux/module.h> > +#include <linux/pm_runtime.h> > +#include <linux/videodev2.h> > +#include <media/v4l2-device.h> > +#include <media/v4l2-event.h> > +#include <media/v4l2-mem2mem.h> > +#include <media/v4l2-ioctl.h> > +#include <media/videobuf2-v4l2.h> > +#include <media/videobuf2-dma-contig.h> > +#include <media/videobuf2-vmalloc.h> > +#include "vpu.h" > +#include "vpu_core.h" > +#include "vpu_v4l2.h" > +#include "vpu_msgs.h" > +#include "vpu_helpers.h" > + > +void vpu_inst_lock(struct vpu_inst *inst) > +{ > + mutex_lock(&inst->lock); > +} > + > +void vpu_inst_unlock(struct vpu_inst *inst) > +{ > + mutex_unlock(&inst->lock); > +} > + > +dma_addr_t vpu_get_vb_phy_addr(struct vb2_buffer *vb, u32 plane_no) > +{ > + if (plane_no >= vb->num_planes) > + return 0; > + return vb2_dma_contig_plane_dma_addr(vb, plane_no) + > + vb->planes[plane_no].data_offset; > +} > + > +unsigned int vpu_get_vb_length(struct vb2_buffer *vb, u32 plane_no) > +{ > + if (plane_no >= vb->num_planes) > + return 0; > + return vb2_plane_size(vb, plane_no) - vb->planes[plane_no].data_offset; > +} > + > +void vpu_set_buffer_state(struct vb2_v4l2_buffer *vbuf, unsigned int state) > +{ > + struct vpu_vb2_buffer *vpu_buf = to_vpu_vb2_buffer(vbuf); > + > + vpu_buf->state = state; > +} > + > +unsigned int vpu_get_buffer_state(struct vb2_v4l2_buffer *vbuf) > +{ > + struct vpu_vb2_buffer *vpu_buf = to_vpu_vb2_buffer(vbuf); > + > + return vpu_buf->state; > +} > + > +void vpu_v4l2_set_error(struct vpu_inst *inst) > +{ > + struct vb2_queue *src_q; > + struct vb2_queue *dst_q; > + > + if (!inst) > + return; > + > + vpu_inst_lock(inst); > + dev_err(inst->dev, "some error occurs in codec\n"); > + if (inst->fh.m2m_ctx) { > + src_q = v4l2_m2m_get_src_vq(inst->fh.m2m_ctx); > + dst_q = v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx); > + if (src_q) > + src_q->error = 1; > + if (dst_q) > + dst_q->error = 1; > + } > + vpu_inst_unlock(inst); > +} > + > +int vpu_notify_eos(struct vpu_inst *inst) > +{ > + static const struct v4l2_event ev = { > + .id = 0, > + .type = V4L2_EVENT_EOS > + }; > + > + vpu_trace(inst->dev, "[%d]\n", inst->id); > + v4l2_event_queue_fh(&inst->fh, &ev); > + > + return 0; > +} > + > +int vpu_notify_source_change(struct vpu_inst *inst) > +{ > + static const struct v4l2_event ev = { > + .id = 0, > + .type = V4L2_EVENT_SOURCE_CHANGE, > + .u.src_change.changes = V4L2_EVENT_SRC_CH_RESOLUTION > + }; > + > + vpu_trace(inst->dev, "[%d]\n", inst->id); > + v4l2_event_queue_fh(&inst->fh, &ev); > + return 0; > +} > + > +int vpu_set_last_buffer_dequeued(struct vpu_inst *inst) > +{ > + struct vb2_queue *q; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + q = v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx); > + if (!list_empty(&q->done_list)) > + return -EINVAL; > + > + if (q->last_buffer_dequeued) > + return 0; > + vpu_trace(inst->dev, "last buffer dequeued\n"); > + q->last_buffer_dequeued = true; > + wake_up(&q->done_wq); > + vpu_notify_eos(inst); > + return 0; > +} > + > +const struct vpu_format *vpu_try_fmt_common(struct vpu_inst *inst, > + struct v4l2_format *f) > +{ > + struct v4l2_pix_format_mplane *pixmp = &f->fmt.pix_mp; > + u32 type = f->type; > + u32 stride = 1; > + u32 bytesperline; > + u32 sizeimage; > + const struct vpu_format *fmt; > + const struct vpu_core_resources *res; > + int i; > + > + fmt = vpu_helper_find_format(inst, type, pixmp->pixelformat); > + if (!fmt) { > + fmt = vpu_helper_enum_format(inst, type, 0); > + if (!fmt) > + return NULL; > + pixmp->pixelformat = fmt->pixfmt; > + } > + > + res = vpu_get_resource(inst); > + if (res) > + stride = res->stride; > + if (pixmp->width) > + pixmp->width = vpu_helper_valid_frame_width(inst, pixmp->width); > + if (pixmp->height) > + pixmp->height = vpu_helper_valid_frame_height(inst, pixmp->height); > + pixmp->flags = fmt->flags; > + pixmp->num_planes = fmt->num_planes; > + if (pixmp->field == V4L2_FIELD_ANY) > + pixmp->field = V4L2_FIELD_NONE; > + for (i = 0; i < pixmp->num_planes; i++) { > + bytesperline = max_t(s32, pixmp->plane_fmt[i].bytesperline, 0); > + sizeimage = vpu_helper_get_plane_size(pixmp->pixelformat, > + pixmp->width, pixmp->height, i, stride, > + pixmp->field > V4L2_FIELD_NONE ? 1 : 0, > + &bytesperline); > + sizeimage = max_t(s32, pixmp->plane_fmt[i].sizeimage, sizeimage); > + pixmp->plane_fmt[i].bytesperline = bytesperline; > + pixmp->plane_fmt[i].sizeimage = sizeimage; > + } > + > + return fmt; > +} > + > +static bool vpu_check_ready(struct vpu_inst *inst, u32 type) > +{ > + if (!inst) > + return false; > + if (inst->state == VPU_CODEC_STATE_DEINIT || inst->id < 0) > + return false; > + if (!inst->ops->check_ready) > + return true; > + return call_vop(inst, check_ready, type); > +} > + > +int vpu_process_output_buffer(struct vpu_inst *inst) > +{ > + struct v4l2_m2m_buffer *buf = NULL; > + struct vb2_v4l2_buffer *vbuf = NULL; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + if (!vpu_check_ready(inst, inst->out_format.type)) > + return -EINVAL; > + > + v4l2_m2m_for_each_src_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vpu_get_buffer_state(vbuf) == VPU_BUF_STATE_IDLE) > + break; > + vbuf = NULL; > + } > + > + if (!vbuf) > + return -EINVAL; > + > + dev_dbg(inst->dev, "[%d]frame id = %d / %d\n", > + inst->id, vbuf->sequence, inst->sequence); > + return call_vop(inst, process_output, &vbuf->vb2_buf); > +} > + > +int vpu_process_capture_buffer(struct vpu_inst *inst) > +{ > + struct v4l2_m2m_buffer *buf = NULL; > + struct vb2_v4l2_buffer *vbuf = NULL; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + if (!vpu_check_ready(inst, inst->cap_format.type)) > + return -EINVAL; > + > + v4l2_m2m_for_each_dst_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vpu_get_buffer_state(vbuf) == VPU_BUF_STATE_IDLE) > + break; > + vbuf = NULL; > + } > + if (!vbuf) > + return -EINVAL; > + > + return call_vop(inst, process_capture, &vbuf->vb2_buf); > +} > + > +struct vb2_v4l2_buffer *vpu_find_buf_by_sequence(struct vpu_inst *inst, > + u32 type, u32 sequence) > +{ > + struct v4l2_m2m_buffer *buf = NULL; > + struct vb2_v4l2_buffer *vbuf = NULL; > + > + if (!inst || !inst->fh.m2m_ctx) > + return NULL; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) { > + v4l2_m2m_for_each_src_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vbuf->sequence == sequence) > + break; > + vbuf = NULL; > + } > + } else { > + v4l2_m2m_for_each_dst_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vbuf->sequence == sequence) > + break; > + vbuf = NULL; > + } > + } > + > + return vbuf; > +} > + > +struct vb2_v4l2_buffer *vpu_find_buf_by_idx(struct vpu_inst *inst, > + u32 type, u32 idx) > +{ > + struct v4l2_m2m_buffer *buf = NULL; > + struct vb2_v4l2_buffer *vbuf = NULL; > + > + if (!inst || !inst->fh.m2m_ctx) > + return NULL; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) { > + v4l2_m2m_for_each_src_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vbuf->vb2_buf.index == idx) > + break; > + vbuf = NULL; > + } > + } else { > + v4l2_m2m_for_each_dst_buf(inst->fh.m2m_ctx, buf) { > + vbuf = &buf->vb; > + if (vbuf->vb2_buf.index == idx) > + break; > + vbuf = NULL; > + } > + } > + > + return vbuf; > +} > + > +int vpu_get_num_buffers(struct vpu_inst *inst, u32 type) > +{ > + struct vb2_queue *q; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) > + q = v4l2_m2m_get_src_vq(inst->fh.m2m_ctx); > + else > + q = v4l2_m2m_get_dst_vq(inst->fh.m2m_ctx); > + > + return q->num_buffers; > +} > + > +static void vpu_m2m_device_run(void *priv) > +{ > +} > + > +static void vpu_m2m_job_abort(void *priv) > +{ > + struct vpu_inst *inst = priv; > + struct v4l2_m2m_ctx *m2m_ctx = inst->fh.m2m_ctx; > + > + v4l2_m2m_job_finish(m2m_ctx->m2m_dev, m2m_ctx); > +} > + > +static const struct v4l2_m2m_ops vpu_m2m_ops = { > + .device_run = vpu_m2m_device_run, > + .job_abort = vpu_m2m_job_abort > +}; > + > +static int vpu_vb2_queue_setup(struct vb2_queue *vq, > + unsigned int *buf_count, > + unsigned int *plane_count, > + unsigned int psize[], > + struct device *allocators[]) > +{ > + struct vpu_inst *inst = vb2_get_drv_priv(vq); > + struct vpu_format *cur_fmt; > + int i; > + > + cur_fmt = vpu_get_format(inst, vq->type); > + > + if (*plane_count) { > + if (inst->state == VPU_CODEC_STATE_DYAMIC_RESOLUTION_CHANGE) { Huh? This 'if' is really weird. Why do you need this? > + for (i = 0; i < *plane_count; i++) { > + if (!psize[i]) This should never happen when you use VIDIOC_CREATE_BUFS to add more buffers. > + psize[i] = cur_fmt->sizeimage[i]; > + } > + return 0; At minimum this needs comments, but it all looks very suspicious. > + } > + if (*plane_count != cur_fmt->num_planes) > + return -EINVAL; > + for (i = 0; i < cur_fmt->num_planes; i++) { > + if (psize[i] < cur_fmt->sizeimage[i]) > + return -EINVAL; > + } > + return 0; > + } > + > + *plane_count = cur_fmt->num_planes; > + for (i = 0; i < cur_fmt->num_planes; i++) > + psize[i] = cur_fmt->sizeimage[i]; > + > + return 0; > +} > + > +static int vpu_vb2_buf_init(struct vb2_buffer *vb) > +{ > + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); > + > + vpu_set_buffer_state(vbuf, VPU_BUF_STATE_IDLE); > + return 0; > +} > + > +static int vpu_vb2_buf_out_validate(struct vb2_buffer *vb) > +{ > + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); > + > + vbuf->field = V4L2_FIELD_NONE; > + > + return 0; > +} > + > +static int vpu_vb2_buf_prepare(struct vb2_buffer *vb) > +{ > + struct vpu_inst *inst = vb2_get_drv_priv(vb->vb2_queue); > + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); > + struct vpu_format *cur_fmt; > + u32 i; > + > + cur_fmt = vpu_get_format(inst, vb->type); > + if (vb->num_planes != cur_fmt->num_planes) > + return -EINVAL; This can't happen, so you can drop this. > + for (i = 0; i < cur_fmt->num_planes; i++) { > + if (vpu_get_vb_length(vb, i) < cur_fmt->sizeimage[i]) { > + dev_dbg(inst->dev, "[%d] %s buf[%d] is invalid\n", > + inst->id, > + vpu_type_name(vb->type), > + vb->index); > + vpu_set_buffer_state(vbuf, VPU_BUF_STATE_ERROR); > + } > + } > + > + return 0; > +} > + > +static void vpu_vb2_buf_finish(struct vb2_buffer *vb) > +{ > + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); > + struct vpu_inst *inst = vb2_get_drv_priv(vb->vb2_queue); > + struct vb2_queue *q = vb->vb2_queue; > + > + if (vbuf->flags & V4L2_BUF_FLAG_LAST) > + vpu_notify_eos(inst); > + > + if (list_empty(&q->done_list)) > + call_vop(inst, on_queue_empty, q->type); > +} > + > +void vpu_vb2_buffers_return(struct vpu_inst *inst, > + unsigned int type, enum vb2_buffer_state state) > +{ > + struct vb2_v4l2_buffer *buf; > + > + if (!inst || !inst->fh.m2m_ctx) > + return; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) { > + while ((buf = v4l2_m2m_src_buf_remove(inst->fh.m2m_ctx))) > + v4l2_m2m_buf_done(buf, state); > + } else { > + while ((buf = v4l2_m2m_dst_buf_remove(inst->fh.m2m_ctx))) > + v4l2_m2m_buf_done(buf, state); > + } > +} > + > +static int vpu_vb2_start_streaming(struct vb2_queue *q, unsigned int count) > +{ > + struct vpu_inst *inst = vb2_get_drv_priv(q); > + struct vpu_format *fmt = vpu_get_format(inst, q->type); > + int ret; > + > + vpu_inst_unlock(inst); > + ret = vpu_inst_register(inst); > + vpu_inst_lock(inst); > + if (ret) { > + vpu_vb2_buffers_return(inst, q->type, VB2_BUF_STATE_QUEUED); > + return ret; > + } > + > + vpu_trace(inst->dev, "[%d] %s %c%c%c%c %dx%d %u(%u) %u(%u) %u(%u) %d\n", > + inst->id, vpu_type_name(q->type), > + fmt->pixfmt, > + fmt->pixfmt >> 8, > + fmt->pixfmt >> 16, > + fmt->pixfmt >> 24, > + fmt->width, fmt->height, > + fmt->sizeimage[0], fmt->bytesperline[0], > + fmt->sizeimage[1], fmt->bytesperline[1], > + fmt->sizeimage[2], fmt->bytesperline[2], > + q->num_buffers); > + call_vop(inst, start, q->type); > + vb2_clear_last_buffer_dequeued(q); > + > + return 0; > +} > + > +static void vpu_vb2_stop_streaming(struct vb2_queue *q) > +{ > + struct vpu_inst *inst = vb2_get_drv_priv(q); > + > + vpu_trace(inst->dev, "[%d] %s\n", inst->id, vpu_type_name(q->type)); > + > + call_vop(inst, stop, q->type); > + vpu_vb2_buffers_return(inst, q->type, VB2_BUF_STATE_ERROR); > + if (V4L2_TYPE_IS_OUTPUT(q->type)) > + inst->sequence = 0; > +} > + > +static void vpu_vb2_buf_queue(struct vb2_buffer *vb) > +{ > + struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb); > + struct vpu_inst *inst = vb2_get_drv_priv(vb->vb2_queue); > + > + if (!inst || !inst->fh.m2m_ctx) > + return; > + if (V4L2_TYPE_IS_OUTPUT(vb->type)) { > + vbuf->sequence = inst->sequence++; > + if ((s64)vb->timestamp < 0) > + vb->timestamp = VPU_INVALID_TIMESTAMP; > + } > + > + v4l2_m2m_buf_queue(inst->fh.m2m_ctx, vbuf); > + vpu_process_output_buffer(inst); > + vpu_process_capture_buffer(inst); > +} > + > +static struct vb2_ops vpu_vb2_ops = { > + .queue_setup = vpu_vb2_queue_setup, > + .buf_init = vpu_vb2_buf_init, > + .buf_out_validate = vpu_vb2_buf_out_validate, > + .buf_prepare = vpu_vb2_buf_prepare, > + .buf_finish = vpu_vb2_buf_finish, > + .start_streaming = vpu_vb2_start_streaming, > + .stop_streaming = vpu_vb2_stop_streaming, > + .buf_queue = vpu_vb2_buf_queue, > + .wait_prepare = vb2_ops_wait_prepare, > + .wait_finish = vb2_ops_wait_finish, > +}; > + > +static int vpu_m2m_queue_init(void *priv, struct vb2_queue *src_vq, > + struct vb2_queue *dst_vq) > +{ > + struct vpu_inst *inst = priv; > + int ret; > + > + inst->out_format.type = src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; > + src_vq->io_modes = VB2_MMAP | VB2_DMABUF; > + src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; > + src_vq->ops = &vpu_vb2_ops; > + src_vq->mem_ops = &vb2_dma_contig_memops; > + if (inst->type == VPU_CORE_TYPE_DEC && inst->use_stream_buffer) > + src_vq->mem_ops = &vb2_vmalloc_memops; > + src_vq->drv_priv = inst; > + src_vq->buf_struct_size = sizeof(struct vpu_vb2_buffer); > + src_vq->min_buffers_needed = 1; > + src_vq->dev = inst->vpu->dev; > + src_vq->lock = &inst->lock; > + ret = vb2_queue_init(src_vq); > + if (ret) > + return ret; > + > + inst->cap_format.type = dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; > + dst_vq->io_modes = VB2_MMAP | VB2_DMABUF; > + dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; > + dst_vq->ops = &vpu_vb2_ops; > + dst_vq->mem_ops = &vb2_dma_contig_memops; > + if (inst->type == VPU_CORE_TYPE_ENC && inst->use_stream_buffer) > + dst_vq->mem_ops = &vb2_vmalloc_memops; > + dst_vq->drv_priv = inst; > + dst_vq->buf_struct_size = sizeof(struct vpu_vb2_buffer); > + dst_vq->min_buffers_needed = 1; > + dst_vq->dev = inst->vpu->dev; > + dst_vq->lock = &inst->lock; > + ret = vb2_queue_init(dst_vq); > + if (ret) { > + vb2_queue_release(src_vq); > + return ret; > + } > + > + return 0; > +} > + > +static int vpu_v4l2_release(struct vpu_inst *inst) > +{ > + vpu_trace(inst->vpu->dev, "%p\n", inst); > + > + vpu_release_core(inst->core); > + put_device(inst->dev); > + > + if (inst->workqueue) { > + cancel_work_sync(&inst->msg_work); > + destroy_workqueue(inst->workqueue); > + inst->workqueue = NULL; > + } > + > + v4l2_ctrl_handler_free(&inst->ctrl_handler); > + mutex_destroy(&inst->lock); > + v4l2_fh_del(&inst->fh); > + v4l2_fh_exit(&inst->fh); > + > + call_vop(inst, cleanup); > + > + return 0; > +} > + > +int vpu_v4l2_open(struct file *file, struct vpu_inst *inst) > +{ > + struct vpu_dev *vpu = video_drvdata(file); > + struct vpu_func *func; > + int ret = 0; > + > + if (!file || !inst || !inst->ops) > + return -EINVAL; > + > + if (inst->type == VPU_CORE_TYPE_ENC) > + func = &vpu->encoder; > + else > + func = &vpu->decoder; > + > + atomic_set(&inst->ref_count, 0); > + vpu_inst_get(inst); > + inst->vpu = vpu; > + inst->core = vpu_request_core(vpu, inst->type); > + if (inst->core) > + inst->dev = get_device(inst->core->dev); > + mutex_init(&inst->lock); > + INIT_LIST_HEAD(&inst->cmd_q); > + inst->id = VPU_INST_NULL_ID; > + inst->release = vpu_v4l2_release; > + inst->pid = current->pid; > + inst->tgid = current->tgid; > + inst->min_buffer_cap = 2; > + inst->min_buffer_out = 2; > + v4l2_fh_init(&inst->fh, func->vfd); > + v4l2_fh_add(&inst->fh); > + > + ret = call_vop(inst, ctrl_init); > + if (ret) > + goto error; > + > + inst->fh.m2m_ctx = v4l2_m2m_ctx_init(func->m2m_dev, > + inst, vpu_m2m_queue_init); > + if (IS_ERR(inst->fh.m2m_ctx)) { > + dev_err(vpu->dev, "v4l2_m2m_ctx_init fail\n"); > + ret = PTR_ERR(func->m2m_dev); > + goto error; > + } > + > + inst->fh.ctrl_handler = &inst->ctrl_handler; > + file->private_data = &inst->fh; > + inst->state = VPU_CODEC_STATE_DEINIT; > + inst->workqueue = alloc_workqueue("vpu_inst", WQ_UNBOUND | WQ_MEM_RECLAIM, 1); > + if (inst->workqueue) { > + INIT_WORK(&inst->msg_work, vpu_inst_run_work); > + ret = kfifo_init(&inst->msg_fifo, > + inst->msg_buffer, > + roundup_pow_of_two(sizeof(inst->msg_buffer))); > + if (ret) { > + destroy_workqueue(inst->workqueue); > + inst->workqueue = NULL; > + } > + } > + vpu_trace(vpu->dev, "tgid = %d, pid = %d, type = %s, inst = %p\n", > + inst->tgid, inst->pid, vpu_core_type_desc(inst->type), inst); > + > + return 0; > +error: > + vpu_inst_put(inst); > + return ret; > +} > + > +int vpu_v4l2_close(struct file *file) > +{ > + struct vpu_dev *vpu = video_drvdata(file); > + struct vpu_inst *inst = to_inst(file); > + > + vpu_trace(vpu->dev, "tgid = %d, pid = %d, inst = %p\n", > + inst->tgid, inst->pid, inst); > + > + vpu_inst_lock(inst); > + if (inst->fh.m2m_ctx) { > + v4l2_m2m_ctx_release(inst->fh.m2m_ctx); > + inst->fh.m2m_ctx = NULL; > + } > + vpu_inst_unlock(inst); > + > + call_vop(inst, release); > + vpu_inst_unregister(inst); > + vpu_inst_put(inst); > + > + return 0; > +} > + > +int vpu_add_func(struct vpu_dev *vpu, struct vpu_func *func) > +{ > + struct video_device *vfd; > + int ret; > + > + if (!vpu || !func) > + return -EINVAL; > + > + if (func->vfd) > + return 0; > + > + func->m2m_dev = v4l2_m2m_init(&vpu_m2m_ops); > + if (IS_ERR(func->m2m_dev)) { > + dev_err(vpu->dev, "v4l2_m2m_init fail\n"); > + func->vfd = NULL; > + return PTR_ERR(func->m2m_dev); > + } > + > + vfd = video_device_alloc(); > + if (!vfd) { > + v4l2_m2m_release(func->m2m_dev); > + dev_err(vpu->dev, "alloc vpu decoder video device fail\n"); > + return -ENOMEM; > + } > + vfd->release = video_device_release; > + vfd->vfl_dir = VFL_DIR_M2M; > + vfd->v4l2_dev = &vpu->v4l2_dev; > + vfd->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING; > + if (func->type == VPU_CORE_TYPE_ENC) { > + strscpy(vfd->name, "amphion-vpu-encoder", sizeof(vfd->name)); > + vfd->fops = venc_get_fops(); > + vfd->ioctl_ops = venc_get_ioctl_ops(); > + } else { > + strscpy(vfd->name, "amphion-vpu-decoder", sizeof(vfd->name)); > + vfd->fops = vdec_get_fops(); > + vfd->ioctl_ops = vdec_get_ioctl_ops(); > + } > + > + ret = video_register_device(vfd, VFL_TYPE_VIDEO, -1); > + if (ret) { > + video_device_release(vfd); > + v4l2_m2m_release(func->m2m_dev); > + return ret; > + } > + video_set_drvdata(vfd, vpu); > + func->vfd = vfd; > + > + ret = v4l2_m2m_register_media_controller(func->m2m_dev, func->vfd, func->function); > + if (ret) { > + v4l2_m2m_release(func->m2m_dev); > + func->m2m_dev = NULL; > + video_unregister_device(func->vfd); > + func->vfd = NULL; > + return ret; > + } > + > + return 0; > +} > + > +void vpu_remove_func(struct vpu_func *func) > +{ > + if (!func) > + return; > + > + if (func->m2m_dev) { > + v4l2_m2m_unregister_media_controller(func->m2m_dev); > + v4l2_m2m_release(func->m2m_dev); > + func->m2m_dev = NULL; > + } > + if (func->vfd) { > + video_unregister_device(func->vfd); > + func->vfd = NULL; > + } > +} > diff --git a/drivers/media/platform/amphion/vpu_v4l2.h b/drivers/media/platform/amphion/vpu_v4l2.h > new file mode 100644 > index 000000000000..17aa36603f82 > --- /dev/null > +++ b/drivers/media/platform/amphion/vpu_v4l2.h > @@ -0,0 +1,56 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#ifndef _AMPHION_VPU_V4L2_H > +#define _AMPHION_VPU_V4L2_H > + > +#include <linux/videodev2.h> > + > +void vpu_inst_lock(struct vpu_inst *inst); > +void vpu_inst_unlock(struct vpu_inst *inst); > +void vpu_set_buffer_state(struct vb2_v4l2_buffer *vbuf, unsigned int state); > +unsigned int vpu_get_buffer_state(struct vb2_v4l2_buffer *vbuf); > + > +int vpu_v4l2_open(struct file *file, struct vpu_inst *inst); > +int vpu_v4l2_close(struct file *file); > + > +const struct vpu_format *vpu_try_fmt_common(struct vpu_inst *inst, struct v4l2_format *f); > +int vpu_process_output_buffer(struct vpu_inst *inst); > +int vpu_process_capture_buffer(struct vpu_inst *inst); > +struct vb2_v4l2_buffer *vpu_find_buf_by_sequence(struct vpu_inst *inst, u32 type, u32 sequence); > +struct vb2_v4l2_buffer *vpu_find_buf_by_idx(struct vpu_inst *inst, u32 type, u32 idx); > +void vpu_v4l2_set_error(struct vpu_inst *inst); > +int vpu_notify_eos(struct vpu_inst *inst); > +int vpu_notify_source_change(struct vpu_inst *inst); > +int vpu_set_last_buffer_dequeued(struct vpu_inst *inst); > +void vpu_vb2_buffers_return(struct vpu_inst *inst, > + unsigned int type, enum vb2_buffer_state state); > +int vpu_get_num_buffers(struct vpu_inst *inst, u32 type); > + > +dma_addr_t vpu_get_vb_phy_addr(struct vb2_buffer *vb, u32 plane_no); > +unsigned int vpu_get_vb_length(struct vb2_buffer *vb, u32 plane_no); > +static inline struct vpu_format *vpu_get_format(struct vpu_inst *inst, u32 type) > +{ > + if (V4L2_TYPE_IS_OUTPUT(type)) > + return &inst->out_format; > + else > + return &inst->cap_format; > +} > + > +static inline char *vpu_type_name(u32 type) > +{ > + return V4L2_TYPE_IS_OUTPUT(type) ? "output" : "capture"; > +} > + > +static inline int vpu_vb_is_codecconfig(struct vb2_v4l2_buffer *vbuf) > +{ > +#ifdef V4L2_BUF_FLAG_CODECCONFIG > + return (vbuf->flags & V4L2_BUF_FLAG_CODECCONFIG) ? 1 : 0; > +#else > + return 0; > +#endif > +} > + > +#endif
Hi Ming, Some comments below: On 12/7/21 09:24, Ming Qian wrote: > This consists of video encoder implementation plus encoder controls. > > Signed-off-by: Ming Qian <ming.qian@nxp.com> > Signed-off-by: Shijie Qin <shijie.qin@nxp.com> > Signed-off-by: Zhou Peng <eagle.zhou@nxp.com> > --- > drivers/media/platform/amphion/venc.c | 1354 +++++++++++++++++++++++++ > 1 file changed, 1354 insertions(+) > create mode 100644 drivers/media/platform/amphion/venc.c > > diff --git a/drivers/media/platform/amphion/venc.c b/drivers/media/platform/amphion/venc.c > new file mode 100644 > index 000000000000..ca394504dbd8 > --- /dev/null > +++ b/drivers/media/platform/amphion/venc.c > @@ -0,0 +1,1354 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright 2020-2021 NXP > + */ > + > +#include <linux/init.h> > +#include <linux/interconnect.h> > +#include <linux/ioctl.h> > +#include <linux/list.h> > +#include <linux/kernel.h> > +#include <linux/module.h> > +#include <linux/delay.h> > +#include <linux/videodev2.h> > +#include <linux/ktime.h> > +#include <linux/rational.h> > +#include <media/v4l2-device.h> > +#include <media/v4l2-event.h> > +#include <media/v4l2-mem2mem.h> > +#include <media/v4l2-ioctl.h> > +#include <media/videobuf2-v4l2.h> > +#include <media/videobuf2-dma-contig.h> > +#include <media/videobuf2-vmalloc.h> > +#include "vpu.h" > +#include "vpu_defs.h" > +#include "vpu_core.h" > +#include "vpu_helpers.h" > +#include "vpu_v4l2.h" > +#include "vpu_cmds.h" > +#include "vpu_rpc.h" > + > +#define VENC_OUTPUT_ENABLE (1 << 0) > +#define VENC_CAPTURE_ENABLE (1 << 1) > +#define VENC_ENABLE_MASK (VENC_OUTPUT_ENABLE | VENC_CAPTURE_ENABLE) > +#define VENC_MAX_BUF_CNT 8 > + > +struct venc_t { > + struct vpu_encode_params params; > + u32 request_key_frame; > + u32 input_ready; > + u32 cpb_size; > + bool bitrate_change; > + > + struct vpu_buffer enc[VENC_MAX_BUF_CNT]; > + struct vpu_buffer ref[VENC_MAX_BUF_CNT]; > + struct vpu_buffer act[VENC_MAX_BUF_CNT]; > + struct list_head frames; > + u32 frame_count; > + u32 encode_count; > + u32 ready_count; > + u32 enable; > + u32 stopped; > + > + u32 skipped_count; > + u32 skipped_bytes; > + > + wait_queue_head_t wq; > +}; > + > +struct venc_frame_t { > + struct list_head list; > + struct vpu_enc_pic_info info; > + u32 bytesused; > + s64 timestamp; > +}; > + > +static const struct vpu_format venc_formats[] = { > + { > + .pixfmt = V4L2_PIX_FMT_NV12M, > + .num_planes = 2, > + .type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, > + }, > + { > + .pixfmt = V4L2_PIX_FMT_H264, > + .num_planes = 1, > + .type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE, > + }, > + {0, 0, 0, 0}, > +}; > + > +static int venc_querycap(struct file *file, void *fh, struct v4l2_capability *cap) > +{ > + strscpy(cap->driver, "amphion-vpu", sizeof(cap->driver)); > + strscpy(cap->card, "amphion vpu encoder", sizeof(cap->card)); > + strscpy(cap->bus_info, "platform: amphion-vpu", sizeof(cap->bus_info)); > + > + return 0; > +} > + > +static int venc_enum_fmt(struct file *file, void *fh, struct v4l2_fmtdesc *f) > +{ > + struct vpu_inst *inst = to_inst(file); > + const struct vpu_format *fmt; > + > + memset(f->reserved, 0, sizeof(f->reserved)); > + fmt = vpu_helper_enum_format(inst, f->type, f->index); > + if (!fmt) > + return -EINVAL; > + > + f->pixelformat = fmt->pixfmt; > + f->flags = fmt->flags; > + > + return 0; > +} > + > +static int venc_enum_framesizes(struct file *file, void *fh, struct v4l2_frmsizeenum *fsize) > +{ > + struct vpu_inst *inst = to_inst(file); > + const struct vpu_core_resources *res; > + > + if (!fsize || fsize->index) > + return -EINVAL; > + > + if (!vpu_helper_find_format(inst, 0, fsize->pixel_format)) > + return -EINVAL; > + > + res = vpu_get_resource(inst); > + if (!res) > + return -EINVAL; > + fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; > + fsize->stepwise.max_width = res->max_width; > + fsize->stepwise.max_height = res->max_height; > + fsize->stepwise.min_width = res->min_width; > + fsize->stepwise.min_height = res->min_height; > + fsize->stepwise.step_width = res->step_width; > + fsize->stepwise.step_height = res->step_height; > + > + return 0; > +} > + > +static int venc_enum_frameintervals(struct file *file, void *fh, struct v4l2_frmivalenum *fival) > +{ > + struct vpu_inst *inst = to_inst(file); > + const struct vpu_core_resources *res; > + > + if (!fival || fival->index) > + return -EINVAL; > + > + if (!vpu_helper_find_format(inst, 0, fival->pixel_format)) > + return -EINVAL; > + > + if (!fival->width || !fival->height) > + return -EINVAL; > + > + res = vpu_get_resource(inst); > + if (!res) > + return -EINVAL; > + if (fival->width < res->min_width || > + fival->width > res->max_width || > + fival->height < res->min_height || > + fival->height > res->max_height) > + return -EINVAL; > + > + fival->type = V4L2_FRMIVAL_TYPE_CONTINUOUS; > + fival->stepwise.min.numerator = 1; > + fival->stepwise.min.denominator = USHRT_MAX; > + fival->stepwise.max.numerator = USHRT_MAX; > + fival->stepwise.max.denominator = 1; > + fival->stepwise.step.numerator = 1; > + fival->stepwise.step.denominator = 1; > + > + return 0; > +} > + > +static int venc_g_fmt(struct file *file, void *fh, struct v4l2_format *f) > +{ > + struct vpu_inst *inst = to_inst(file); > + struct venc_t *venc = inst->priv; > + struct v4l2_pix_format_mplane *pixmp = &f->fmt.pix_mp; > + struct vpu_format *cur_fmt; > + int i; > + > + cur_fmt = vpu_get_format(inst, f->type); > + > + pixmp->pixelformat = cur_fmt->pixfmt; > + pixmp->num_planes = cur_fmt->num_planes; > + pixmp->width = cur_fmt->width; > + pixmp->height = cur_fmt->height; > + pixmp->field = cur_fmt->field; > + pixmp->flags = cur_fmt->flags; > + for (i = 0; i < pixmp->num_planes; i++) { > + pixmp->plane_fmt[i].bytesperline = cur_fmt->bytesperline[i]; > + pixmp->plane_fmt[i].sizeimage = cur_fmt->sizeimage[i]; > + } > + > + f->fmt.pix_mp.colorspace = venc->params.color.primaries; > + f->fmt.pix_mp.xfer_func = venc->params.color.transfer; > + f->fmt.pix_mp.ycbcr_enc = venc->params.color.matrix; > + f->fmt.pix_mp.quantization = venc->params.color.full_range; > + > + return 0; > +} > + > +static int venc_try_fmt(struct file *file, void *fh, struct v4l2_format *f) > +{ > + struct vpu_inst *inst = to_inst(file); > + > + vpu_try_fmt_common(inst, f); > + > + return 0; > +} > + > +static int venc_s_fmt(struct file *file, void *fh, struct v4l2_format *f) > +{ > + struct vpu_inst *inst = to_inst(file); > + const struct vpu_format *fmt; > + struct vpu_format *cur_fmt; > + struct vb2_queue *q; > + struct venc_t *venc = inst->priv; > + struct v4l2_pix_format_mplane *pix_mp = &f->fmt.pix_mp; > + int i; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + q = v4l2_m2m_get_vq(inst->fh.m2m_ctx, f->type); > + if (!q) > + return -EINVAL; > + if (vb2_is_streaming(q)) That should be vb2_is_busy(): once buffers have been allocated you can no longer change the format. Same issue is present in the next patch. Regards, Hans > + return -EBUSY; > + > + fmt = vpu_try_fmt_common(inst, f); > + if (!fmt) > + return -EINVAL; > + > + cur_fmt = vpu_get_format(inst, f->type); > + > + cur_fmt->pixfmt = fmt->pixfmt; > + cur_fmt->num_planes = fmt->num_planes; > + cur_fmt->flags = fmt->flags; > + cur_fmt->width = pix_mp->width; > + cur_fmt->height = pix_mp->height; > + for (i = 0; i < fmt->num_planes; i++) { > + cur_fmt->sizeimage[i] = pix_mp->plane_fmt[i].sizeimage; > + cur_fmt->bytesperline[i] = pix_mp->plane_fmt[i].bytesperline; > + } > + > + if (pix_mp->field != V4L2_FIELD_ANY) > + cur_fmt->field = pix_mp->field; > + > + if (V4L2_TYPE_IS_OUTPUT(f->type)) { > + venc->params.input_format = cur_fmt->pixfmt; > + venc->params.src_stride = cur_fmt->bytesperline[0]; > + venc->params.src_width = cur_fmt->width; > + venc->params.src_height = cur_fmt->height; > + venc->params.crop.left = 0; > + venc->params.crop.top = 0; > + venc->params.crop.width = cur_fmt->width; > + venc->params.crop.height = cur_fmt->height; > + } else { > + venc->params.codec_format = cur_fmt->pixfmt; > + venc->params.out_width = cur_fmt->width; > + venc->params.out_height = cur_fmt->height; > + } > + > + if (V4L2_TYPE_IS_OUTPUT(f->type)) { > + if (!vpu_color_check_primaries(pix_mp->colorspace)) { > + venc->params.color.primaries = pix_mp->colorspace; > + vpu_color_get_default(venc->params.color.primaries, > + &venc->params.color.transfer, > + &venc->params.color.matrix, > + &venc->params.color.full_range); > + } > + if (!vpu_color_check_transfers(pix_mp->xfer_func)) > + venc->params.color.transfer = pix_mp->xfer_func; > + if (!vpu_color_check_matrix(pix_mp->ycbcr_enc)) > + venc->params.color.matrix = pix_mp->ycbcr_enc; > + if (!vpu_color_check_full_range(pix_mp->quantization)) > + venc->params.color.full_range = pix_mp->quantization; > + } > + > + pix_mp->colorspace = venc->params.color.primaries; > + pix_mp->xfer_func = venc->params.color.transfer; > + pix_mp->ycbcr_enc = venc->params.color.matrix; > + pix_mp->quantization = venc->params.color.full_range; > + > + return 0; > +} > + > +static int venc_g_parm(struct file *file, void *fh, struct v4l2_streamparm *parm) > +{ > + struct vpu_inst *inst = to_inst(file); > + struct venc_t *venc = inst->priv; > + struct v4l2_fract *timeperframe = &parm->parm.capture.timeperframe; > + > + if (!parm) > + return -EINVAL; > + > + if (!vpu_helper_check_type(inst, parm->type)) > + return -EINVAL; > + > + parm->parm.capture.capability = V4L2_CAP_TIMEPERFRAME; > + parm->parm.capture.readbuffers = 0; > + timeperframe->numerator = venc->params.frame_rate.numerator; > + timeperframe->denominator = venc->params.frame_rate.denominator; > + > + return 0; > +} > + > +static int venc_s_parm(struct file *file, void *fh, struct v4l2_streamparm *parm) > +{ > + struct vpu_inst *inst = to_inst(file); > + struct venc_t *venc = inst->priv; > + struct v4l2_fract *timeperframe = &parm->parm.capture.timeperframe; > + unsigned long n, d; > + > + if (!parm) > + return -EINVAL; > + > + if (!vpu_helper_check_type(inst, parm->type)) > + return -EINVAL; > + > + if (!timeperframe->numerator) > + timeperframe->numerator = venc->params.frame_rate.numerator; > + if (!timeperframe->denominator) > + timeperframe->denominator = venc->params.frame_rate.denominator; > + > + venc->params.frame_rate.numerator = timeperframe->numerator; > + venc->params.frame_rate.denominator = timeperframe->denominator; > + > + rational_best_approximation(venc->params.frame_rate.numerator, > + venc->params.frame_rate.denominator, > + venc->params.frame_rate.numerator, > + venc->params.frame_rate.denominator, > + &n, &d); > + venc->params.frame_rate.numerator = n; > + venc->params.frame_rate.denominator = d; > + > + parm->parm.capture.capability = V4L2_CAP_TIMEPERFRAME; > + memset(parm->parm.capture.reserved, > + 0, sizeof(parm->parm.capture.reserved)); > + > + return 0; > +} > + > +static int venc_g_selection(struct file *file, void *fh, struct v4l2_selection *s) > +{ > + struct vpu_inst *inst = to_inst(file); > + struct venc_t *venc = inst->priv; > + > + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT && s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) > + return -EINVAL; > + > + switch (s->target) { > + case V4L2_SEL_TGT_CROP_DEFAULT: > + case V4L2_SEL_TGT_CROP_BOUNDS: > + s->r.left = 0; > + s->r.top = 0; > + s->r.width = inst->out_format.width; > + s->r.height = inst->out_format.height; > + break; > + case V4L2_SEL_TGT_CROP: > + s->r = venc->params.crop; > + break; > + default: > + return -EINVAL; > + } > + > + return 0; > +} > + > +static int venc_valid_crop(struct venc_t *venc, const struct vpu_core_resources *res) > +{ > + struct v4l2_rect *rect = NULL; > + u32 min_width; > + u32 min_height; > + u32 src_width; > + u32 src_height; > + > + rect = &venc->params.crop; > + min_width = res->min_width; > + min_height = res->min_height; > + src_width = venc->params.src_width; > + src_height = venc->params.src_height; > + > + if (rect->width == 0 || rect->height == 0) > + return -EINVAL; > + if (rect->left > src_width - min_width || > + rect->top > src_height - min_height) > + return -EINVAL; > + > + rect->width = min(rect->width, src_width - rect->left); > + rect->width = max_t(u32, rect->width, min_width); > + > + rect->height = min(rect->height, src_height - rect->top); > + rect->height = max_t(u32, rect->height, min_height); > + > + return 0; > +} > + > +static int venc_s_selection(struct file *file, void *fh, struct v4l2_selection *s) > +{ > + struct vpu_inst *inst = to_inst(file); > + const struct vpu_core_resources *res; > + struct venc_t *venc = inst->priv; > + > + res = vpu_get_resource(inst); > + if (!res) > + return -EINVAL; > + > + if (s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT && s->type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) > + return -EINVAL; > + if (s->target != V4L2_SEL_TGT_CROP) > + return -EINVAL; > + > + venc->params.crop.left = ALIGN(s->r.left, res->step_width); > + venc->params.crop.top = ALIGN(s->r.top, res->step_height); > + venc->params.crop.width = ALIGN(s->r.width, res->step_width); > + venc->params.crop.height = ALIGN(s->r.height, res->step_height); > + if (venc_valid_crop(venc, res)) { > + venc->params.crop.left = 0; > + venc->params.crop.top = 0; > + venc->params.crop.width = venc->params.src_width; > + venc->params.crop.height = venc->params.src_height; > + } > + > + inst->crop = venc->params.crop; > + > + return 0; > +} > + > +static int venc_drain(struct vpu_inst *inst) > +{ > + struct venc_t *venc = inst->priv; > + int ret; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + if (inst->state != VPU_CODEC_STATE_DRAIN) > + return 0; > + > + if (v4l2_m2m_num_src_bufs_ready(inst->fh.m2m_ctx)) > + return 0; > + > + if (!venc->input_ready) > + return 0; > + > + venc->input_ready = false; > + vpu_trace(inst->dev, "[%d]\n", inst->id); > + ret = vpu_session_stop(inst); > + if (ret) > + return ret; > + inst->state = VPU_CODEC_STATE_STOP; > + wake_up_all(&venc->wq); > + > + return 0; > +} > + > +static int venc_request_eos(struct vpu_inst *inst) > +{ > + inst->state = VPU_CODEC_STATE_DRAIN; > + venc_drain(inst); > + > + return 0; > +} > + > +static int venc_encoder_cmd(struct file *file, void *fh, struct v4l2_encoder_cmd *cmd) > +{ > + struct vpu_inst *inst = to_inst(file); > + int ret; > + > + ret = v4l2_m2m_ioctl_try_encoder_cmd(file, fh, cmd); > + if (ret) > + return ret; > + > + vpu_inst_lock(inst); > + if (cmd->cmd == V4L2_ENC_CMD_STOP) { > + if (inst->state == VPU_CODEC_STATE_DEINIT) > + vpu_set_last_buffer_dequeued(inst); > + else > + venc_request_eos(inst); > + } > + vpu_inst_unlock(inst); > + > + return 0; > +} > + > +static int venc_subscribe_event(struct v4l2_fh *fh, const struct v4l2_event_subscription *sub) > +{ > + switch (sub->type) { > + case V4L2_EVENT_EOS: > + return v4l2_event_subscribe(fh, sub, 0, NULL); > + case V4L2_EVENT_CTRL: > + return v4l2_ctrl_subscribe_event(fh, sub); > + default: > + return -EINVAL; > + } > +} > + > +static const struct v4l2_ioctl_ops venc_ioctl_ops = { > + .vidioc_querycap = venc_querycap, > + .vidioc_enum_fmt_vid_cap = venc_enum_fmt, > + .vidioc_enum_fmt_vid_out = venc_enum_fmt, > + .vidioc_enum_framesizes = venc_enum_framesizes, > + .vidioc_enum_frameintervals = venc_enum_frameintervals, > + .vidioc_g_fmt_vid_cap_mplane = venc_g_fmt, > + .vidioc_g_fmt_vid_out_mplane = venc_g_fmt, > + .vidioc_try_fmt_vid_cap_mplane = venc_try_fmt, > + .vidioc_try_fmt_vid_out_mplane = venc_try_fmt, > + .vidioc_s_fmt_vid_cap_mplane = venc_s_fmt, > + .vidioc_s_fmt_vid_out_mplane = venc_s_fmt, > + .vidioc_g_parm = venc_g_parm, > + .vidioc_s_parm = venc_s_parm, > + .vidioc_g_selection = venc_g_selection, > + .vidioc_s_selection = venc_s_selection, > + .vidioc_try_encoder_cmd = v4l2_m2m_ioctl_try_encoder_cmd, > + .vidioc_encoder_cmd = venc_encoder_cmd, > + .vidioc_subscribe_event = venc_subscribe_event, > + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, > + .vidioc_reqbufs = v4l2_m2m_ioctl_reqbufs, > + .vidioc_querybuf = v4l2_m2m_ioctl_querybuf, > + .vidioc_create_bufs = v4l2_m2m_ioctl_create_bufs, > + .vidioc_prepare_buf = v4l2_m2m_ioctl_prepare_buf, > + .vidioc_qbuf = v4l2_m2m_ioctl_qbuf, > + .vidioc_expbuf = v4l2_m2m_ioctl_expbuf, > + .vidioc_dqbuf = v4l2_m2m_ioctl_dqbuf, > + .vidioc_streamon = v4l2_m2m_ioctl_streamon, > + .vidioc_streamoff = v4l2_m2m_ioctl_streamoff, > +}; > + > +static int venc_op_s_ctrl(struct v4l2_ctrl *ctrl) > +{ > + struct vpu_inst *inst = ctrl_to_inst(ctrl); > + struct venc_t *venc = inst->priv; > + int ret = 0; > + > + vpu_inst_lock(inst); > + switch (ctrl->id) { > + case V4L2_CID_MPEG_VIDEO_H264_PROFILE: > + venc->params.profile = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_LEVEL: > + venc->params.level = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_BITRATE_MODE: > + venc->params.rc_mode = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_BITRATE: > + if (ctrl->val != venc->params.bitrate) > + venc->bitrate_change = true; > + venc->params.bitrate = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_GOP_SIZE: > + venc->params.gop_length = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_B_FRAMES: > + venc->params.bframes = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_I_FRAME_QP: > + venc->params.i_frame_qp = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_P_FRAME_QP: > + venc->params.p_frame_qp = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_B_FRAME_QP: > + venc->params.b_frame_qp = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME: > + venc->request_key_frame = 1; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_CPB_SIZE: > + venc->cpb_size = ctrl->val * 1024; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_ENABLE: > + venc->params.sar.enable = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_IDC: > + venc->params.sar.idc = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_VUI_EXT_SAR_WIDTH: > + venc->params.sar.width = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_H264_VUI_EXT_SAR_HEIGHT: > + venc->params.sar.height = ctrl->val; > + break; > + case V4L2_CID_MPEG_VIDEO_HEADER_MODE: > + break; > + default: > + ret = -EINVAL; > + break; > + } > + vpu_inst_unlock(inst); > + > + return ret; > +} > + > +static const struct v4l2_ctrl_ops venc_ctrl_ops = { > + .s_ctrl = venc_op_s_ctrl, > + .g_volatile_ctrl = vpu_helper_g_volatile_ctrl, > +}; > + > +static int venc_ctrl_init(struct vpu_inst *inst) > +{ > + struct v4l2_ctrl *ctrl; > + int ret; > + > + ret = v4l2_ctrl_handler_init(&inst->ctrl_handler, 20); > + if (ret) > + return ret; > + > + v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_PROFILE, > + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH, > + ~((1 << V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE) | > + (1 << V4L2_MPEG_VIDEO_H264_PROFILE_MAIN) | > + (1 << V4L2_MPEG_VIDEO_H264_PROFILE_HIGH)), > + V4L2_MPEG_VIDEO_H264_PROFILE_HIGH); > + > + v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_LEVEL, > + V4L2_MPEG_VIDEO_H264_LEVEL_5_1, > + 0x0, > + V4L2_MPEG_VIDEO_H264_LEVEL_4_0); > + > + v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_BITRATE_MODE, > + V4L2_MPEG_VIDEO_BITRATE_MODE_CBR, > + 0x0, > + V4L2_MPEG_VIDEO_BITRATE_MODE_CBR); > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_BITRATE, > + BITRATE_MIN, > + BITRATE_MAX, > + BITRATE_STEP, > + BITRATE_DEFAULT); > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_GOP_SIZE, 0, (1 << 16) - 1, 1, 30); > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_B_FRAMES, 0, 4, 1, 0); > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_I_FRAME_QP, 1, 51, 1, 26); > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_P_FRAME_QP, 1, 51, 1, 28); > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_B_FRAME_QP, 1, 51, 1, 30); > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME, 0, 0, 0, 0); > + ctrl = v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MIN_BUFFERS_FOR_CAPTURE, 1, 32, 1, 2); > + if (ctrl) > + ctrl->flags |= V4L2_CTRL_FLAG_VOLATILE; > + ctrl = v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MIN_BUFFERS_FOR_OUTPUT, 1, 32, 1, 2); > + if (ctrl) > + ctrl->flags |= V4L2_CTRL_FLAG_VOLATILE; > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_CPB_SIZE, 64, 10240, 1, 1024); > + > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_ENABLE, 0, 1, 1, 1); > + v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_IDC, > + V4L2_MPEG_VIDEO_H264_VUI_SAR_IDC_EXTENDED, > + 0x0, > + V4L2_MPEG_VIDEO_H264_VUI_SAR_IDC_1x1); > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_VUI_EXT_SAR_WIDTH, > + 0, USHRT_MAX, 1, 1); > + v4l2_ctrl_new_std(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_H264_VUI_EXT_SAR_HEIGHT, > + 0, USHRT_MAX, 1, 1); > + v4l2_ctrl_new_std_menu(&inst->ctrl_handler, &venc_ctrl_ops, > + V4L2_CID_MPEG_VIDEO_HEADER_MODE, > + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME, > + ~(1 << V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME), > + V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME); > + > + ret = v4l2_ctrl_handler_setup(&inst->ctrl_handler); > + if (ret) { > + dev_err(inst->dev, "[%d] setup ctrls fail, ret = %d\n", inst->id, ret); > + v4l2_ctrl_handler_free(&inst->ctrl_handler); > + return ret; > + } > + > + return 0; > +} > + > +static bool venc_check_ready(struct vpu_inst *inst, unsigned int type) > +{ > + struct venc_t *venc = inst->priv; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) { > + if (vpu_helper_get_free_space(inst) < venc->cpb_size) > + return false; > + return venc->input_ready; > + } > + > + if (list_empty(&venc->frames)) > + return false; > + return true; > +} > + > +static u32 venc_get_enable_mask(u32 type) > +{ > + if (V4L2_TYPE_IS_OUTPUT(type)) > + return VENC_OUTPUT_ENABLE; > + else > + return VENC_CAPTURE_ENABLE; > +} > + > +static void venc_set_enable(struct venc_t *venc, u32 type, int enable) > +{ > + u32 mask = venc_get_enable_mask(type); > + > + if (enable) > + venc->enable |= mask; > + else > + venc->enable &= ~mask; > +} > + > +static u32 venc_get_enable(struct venc_t *venc, u32 type) > +{ > + return venc->enable & venc_get_enable_mask(type); > +} > + > +static void venc_input_done(struct vpu_inst *inst) > +{ > + struct venc_t *venc = inst->priv; > + > + vpu_inst_lock(inst); > + venc->input_ready = true; > + vpu_process_output_buffer(inst); > + if (inst->state == VPU_CODEC_STATE_DRAIN) > + venc_drain(inst); > + vpu_inst_unlock(inst); > +} > + > +/* > + * It's hardware limitation, that there may be several bytes > + * redundant data at the beginning of frame. > + * For android platform, the redundant data may cause cts test fail > + * So driver will strip them > + */ > +static int venc_precheck_encoded_frame(struct vpu_inst *inst, struct venc_frame_t *frame) > +{ > + struct venc_t *venc; > + int skipped; > + > + if (!inst || !frame || !frame->bytesused) > + return -EINVAL; > + > + venc = inst->priv; > + skipped = vpu_helper_find_startcode(&inst->stream_buffer, > + inst->cap_format.pixfmt, > + frame->info.wptr - inst->stream_buffer.phys, > + frame->bytesused); > + if (skipped > 0) { > + frame->bytesused -= skipped; > + frame->info.wptr = vpu_helper_step_walk(&inst->stream_buffer, > + frame->info.wptr, skipped); > + venc->skipped_bytes += skipped; > + venc->skipped_count++; > + } > + > + return 0; > +} > + > +static int venc_get_one_encoded_frame(struct vpu_inst *inst, > + struct venc_frame_t *frame, > + struct vb2_v4l2_buffer *vbuf) > +{ > + struct venc_t *venc = inst->priv; > + > + if (!vbuf) > + return -EAGAIN; > + > + if (!venc_get_enable(inst->priv, vbuf->vb2_buf.type)) { > + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR); > + return 0; > + } > + if (frame->bytesused > vbuf->vb2_buf.planes[0].length) { > + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR); > + return -ENOMEM; > + } > + > + venc_precheck_encoded_frame(inst, frame); > + > + if (frame->bytesused) { > + u32 rptr = frame->info.wptr; > + void *dst = vb2_plane_vaddr(&vbuf->vb2_buf, 0); > + > + vpu_helper_copy_from_stream_buffer(&inst->stream_buffer, > + &rptr, frame->bytesused, dst); > + vpu_iface_update_stream_buffer(inst, rptr, 0); > + } > + vb2_set_plane_payload(&vbuf->vb2_buf, 0, frame->bytesused); > + vbuf->sequence = frame->info.frame_id; > + vbuf->vb2_buf.timestamp = frame->info.timestamp; > + vbuf->field = inst->cap_format.field; > + vbuf->flags |= frame->info.pic_type; > + vpu_set_buffer_state(vbuf, VPU_BUF_STATE_IDLE); > + dev_dbg(inst->dev, "[%d][OUTPUT TS]%32lld\n", inst->id, frame->info.timestamp); > + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_DONE); > + venc->ready_count++; > + > + if (vbuf->flags & V4L2_BUF_FLAG_KEYFRAME) > + dev_dbg(inst->dev, "[%d][%d]key frame\n", inst->id, frame->info.frame_id); > + > + return 0; > +} > + > +static int venc_get_encoded_frames(struct vpu_inst *inst) > +{ > + struct venc_t *venc; > + struct venc_frame_t *frame; > + struct venc_frame_t *tmp; > + > + if (!inst || !inst->priv || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + venc = inst->priv; > + list_for_each_entry_safe(frame, tmp, &venc->frames, list) { > + if (venc_get_one_encoded_frame(inst, frame, > + v4l2_m2m_dst_buf_remove(inst->fh.m2m_ctx))) > + break; > + list_del_init(&frame->list); > + vfree(frame); > + } > + > + return 0; > +} > + > +static int venc_frame_encoded(struct vpu_inst *inst, void *arg) > +{ > + struct vpu_enc_pic_info *info = arg; > + struct venc_frame_t *frame; > + struct venc_t *venc; > + int ret = 0; > + > + if (!inst || !info) > + return -EINVAL; > + venc = inst->priv; > + frame = vzalloc(sizeof(*frame)); > + if (!frame) > + return -ENOMEM; > + > + memcpy(&frame->info, info, sizeof(frame->info)); > + frame->bytesused = info->frame_size; > + > + vpu_inst_lock(inst); > + list_add_tail(&frame->list, &venc->frames); > + venc->encode_count++; > + venc_get_encoded_frames(inst); > + vpu_inst_unlock(inst); > + > + return ret; > +} > + > +static void venc_buf_done(struct vpu_inst *inst, struct vpu_frame_info *frame) > +{ > + struct vb2_v4l2_buffer *vbuf; > + > + if (!inst || !frame || !inst->fh.m2m_ctx) > + return; > + > + vpu_inst_lock(inst); > + if (!venc_get_enable(inst->priv, frame->type)) > + goto exit; > + vbuf = vpu_find_buf_by_sequence(inst, frame->type, frame->sequence); > + if (!vbuf) { > + dev_err(inst->dev, "[%d] can't find buf: type %d, sequence %d\n", > + inst->id, frame->type, frame->sequence); > + goto exit; > + } > + > + vpu_set_buffer_state(vbuf, VPU_BUF_STATE_IDLE); > + if (V4L2_TYPE_IS_OUTPUT(frame->type)) > + v4l2_m2m_src_buf_remove_by_buf(inst->fh.m2m_ctx, vbuf); > + else > + v4l2_m2m_dst_buf_remove_by_buf(inst->fh.m2m_ctx, vbuf); > + v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_DONE); > +exit: > + vpu_inst_unlock(inst); > +} > + > +static void venc_set_last_buffer_dequeued(struct vpu_inst *inst) > +{ > + struct venc_t *venc = inst->priv; > + > + if (venc->stopped && list_empty(&venc->frames)) > + vpu_set_last_buffer_dequeued(inst); > +} > + > +static void venc_stop_done(struct vpu_inst *inst) > +{ > + struct venc_t *venc = inst->priv; > + > + vpu_inst_lock(inst); > + venc->stopped = true; > + venc_set_last_buffer_dequeued(inst); > + vpu_inst_unlock(inst); > + > + wake_up_all(&venc->wq); > +} > + > +static void venc_event_notify(struct vpu_inst *inst, u32 event, void *data) > +{ > +} > + > +static void venc_release(struct vpu_inst *inst) > +{ > +} > + > +static void venc_cleanup(struct vpu_inst *inst) > +{ > + struct venc_t *venc; > + > + if (!inst) > + return; > + > + venc = inst->priv; > + if (venc) > + vfree(venc); > + inst->priv = NULL; > + vfree(inst); > +} > + > +static int venc_start_session(struct vpu_inst *inst, u32 type) > +{ > + struct venc_t *venc = inst->priv; > + int stream_buffer_size; > + int ret; > + > + venc_set_enable(venc, type, 1); > + if ((venc->enable & VENC_ENABLE_MASK) != VENC_ENABLE_MASK) > + return 0; > + > + vpu_iface_init_instance(inst); > + stream_buffer_size = vpu_iface_get_stream_buffer_size(inst->core); > + if (stream_buffer_size > 0) { > + inst->stream_buffer.length = max_t(u32, stream_buffer_size, venc->cpb_size * 3); > + ret = vpu_alloc_dma(inst->core, &inst->stream_buffer); > + if (ret) > + goto error; > + > + inst->use_stream_buffer = true; > + vpu_iface_config_stream_buffer(inst, &inst->stream_buffer); > + } > + > + ret = vpu_iface_set_encode_params(inst, &venc->params, 0); > + if (ret) > + goto error; > + ret = vpu_session_configure_codec(inst); > + if (ret) > + goto error; > + > + inst->state = VPU_CODEC_STATE_CONFIGURED; > + /*vpu_iface_config_memory_resource*/ > + > + /*config enc expert mode parameter*/ > + ret = vpu_iface_set_encode_params(inst, &venc->params, 1); > + if (ret) > + goto error; > + > + ret = vpu_session_start(inst); > + if (ret) > + goto error; > + inst->state = VPU_CODEC_STATE_STARTED; > + > + venc->bitrate_change = false; > + venc->input_ready = true; > + venc->frame_count = 0; > + venc->encode_count = 0; > + venc->ready_count = 0; > + venc->stopped = false; > + vpu_process_output_buffer(inst); > + if (venc->frame_count == 0) > + dev_err(inst->dev, "[%d] there is no input when starting\n", inst->id); > + > + return 0; > +error: > + venc_set_enable(venc, type, 0); > + inst->state = VPU_CODEC_STATE_DEINIT; > + > + vpu_free_dma(&inst->stream_buffer); > + return ret; > +} > + > +static void venc_cleanup_mem_resource(struct vpu_inst *inst) > +{ > + struct venc_t *venc; > + u32 i; > + > + venc = inst->priv; > + > + for (i = 0; i < ARRAY_SIZE(venc->enc); i++) > + vpu_free_dma(&venc->enc[i]); > + for (i = 0; i < ARRAY_SIZE(venc->ref); i++) > + vpu_free_dma(&venc->ref[i]); > + for (i = 0; i < ARRAY_SIZE(venc->act); i++) > + vpu_free_dma(&venc->act[i]); > +} > + > +static void venc_request_mem_resource(struct vpu_inst *inst, > + u32 enc_frame_size, > + u32 enc_frame_num, > + u32 ref_frame_size, > + u32 ref_frame_num, > + u32 act_frame_size, > + u32 act_frame_num) > +{ > + struct venc_t *venc; > + u32 i; > + int ret; > + > + venc = inst->priv; > + if (enc_frame_num > ARRAY_SIZE(venc->enc)) { > + dev_err(inst->dev, "[%d] enc num(%d) is out of range\n", > + inst->id, enc_frame_num); > + return; > + } > + if (ref_frame_num > ARRAY_SIZE(venc->ref)) { > + dev_err(inst->dev, "[%d] ref num(%d) is out of range\n", > + inst->id, ref_frame_num); > + return; > + } > + if (act_frame_num > ARRAY_SIZE(venc->act)) { > + dev_err(inst->dev, "[%d] act num(%d) is out of range\n", > + inst->id, act_frame_num); > + return; > + } > + > + for (i = 0; i < enc_frame_num; i++) { > + venc->enc[i].length = enc_frame_size; > + ret = vpu_alloc_dma(inst->core, &venc->enc[i]); > + if (ret) { > + venc_cleanup_mem_resource(inst); > + return; > + } > + } > + for (i = 0; i < ref_frame_num; i++) { > + venc->ref[i].length = ref_frame_size; > + ret = vpu_alloc_dma(inst->core, &venc->ref[i]); > + if (ret) { > + venc_cleanup_mem_resource(inst); > + return; > + } > + } > + if (act_frame_num != 1 || act_frame_size > inst->act.length) { > + venc_cleanup_mem_resource(inst); > + return; > + } > + venc->act[0].length = act_frame_size; > + venc->act[0].phys = inst->act.phys; > + venc->act[0].virt = inst->act.virt; > + > + for (i = 0; i < enc_frame_num; i++) > + vpu_iface_config_memory_resource(inst, MEM_RES_ENC, i, &venc->enc[i]); > + for (i = 0; i < ref_frame_num; i++) > + vpu_iface_config_memory_resource(inst, MEM_RES_REF, i, &venc->ref[i]); > + for (i = 0; i < act_frame_num; i++) > + vpu_iface_config_memory_resource(inst, MEM_RES_ACT, i, &venc->act[i]); > +} > + > +static void venc_cleanup_frames(struct venc_t *venc) > +{ > + struct venc_frame_t *frame; > + struct venc_frame_t *tmp; > + > + list_for_each_entry_safe(frame, tmp, &venc->frames, list) { > + list_del_init(&frame->list); > + vfree(frame); > + } > +} > + > +static int venc_stop_session(struct vpu_inst *inst, u32 type) > +{ > + struct venc_t *venc = inst->priv; > + > + venc_set_enable(venc, type, 0); > + if (venc->enable & VENC_ENABLE_MASK) > + return 0; > + > + if (inst->state == VPU_CODEC_STATE_DEINIT) > + return 0; > + > + if (inst->state != VPU_CODEC_STATE_STOP) > + venc_request_eos(inst); > + > + call_vop(inst, wait_prepare); > + if (!wait_event_timeout(venc->wq, venc->stopped, VPU_TIMEOUT)) { > + set_bit(inst->id, &inst->core->hang_mask); > + vpu_session_debug(inst); > + } > + call_vop(inst, wait_finish); > + > + inst->state = VPU_CODEC_STATE_DEINIT; > + venc_cleanup_frames(inst->priv); > + vpu_free_dma(&inst->stream_buffer); > + venc_cleanup_mem_resource(inst); > + > + return 0; > +} > + > +static int venc_process_output(struct vpu_inst *inst, struct vb2_buffer *vb) > +{ > + struct venc_t *venc = inst->priv; > + struct vb2_v4l2_buffer *vbuf; > + u32 flags; > + > + if (inst->state == VPU_CODEC_STATE_DEINIT) > + return -EINVAL; > + > + vbuf = to_vb2_v4l2_buffer(vb); > + if (inst->state == VPU_CODEC_STATE_STARTED) > + inst->state = VPU_CODEC_STATE_ACTIVE; > + > + flags = vbuf->flags; > + if (venc->request_key_frame) { > + vbuf->flags |= V4L2_BUF_FLAG_KEYFRAME; > + venc->request_key_frame = 0; > + } > + if (venc->bitrate_change) { > + vpu_session_update_parameters(inst, &venc->params); > + venc->bitrate_change = false; > + } > + dev_dbg(inst->dev, "[%d][INPUT TS]%32lld\n", inst->id, vb->timestamp); > + vpu_iface_input_frame(inst, vb); > + vbuf->flags = flags; > + venc->input_ready = false; > + venc->frame_count++; > + vpu_set_buffer_state(vbuf, VPU_BUF_STATE_INUSE); > + > + return 0; > +} > + > +static int venc_process_capture(struct vpu_inst *inst, struct vb2_buffer *vb) > +{ > + struct venc_t *venc; > + struct venc_frame_t *frame = NULL; > + struct vb2_v4l2_buffer *vbuf; > + int ret; > + > + if (!inst || !inst->fh.m2m_ctx) > + return -EINVAL; > + > + venc = inst->priv; > + if (list_empty(&venc->frames)) > + return -EINVAL; > + > + frame = list_first_entry(&venc->frames, struct venc_frame_t, list); > + vbuf = to_vb2_v4l2_buffer(vb); > + v4l2_m2m_dst_buf_remove_by_buf(inst->fh.m2m_ctx, vbuf); > + ret = venc_get_one_encoded_frame(inst, frame, vbuf); > + if (ret) > + return ret; > + > + list_del_init(&frame->list); > + vfree(frame); > + return 0; > +} > + > +static void venc_on_queue_empty(struct vpu_inst *inst, u32 type) > +{ > + struct venc_t *venc = inst->priv; > + > + if (V4L2_TYPE_IS_OUTPUT(type)) > + return; > + > + if (venc->stopped) > + venc_set_last_buffer_dequeued(inst); > +} > + > +static int venc_get_debug_info(struct vpu_inst *inst, char *str, u32 size, u32 i) > +{ > + struct venc_t *venc = inst->priv; > + int num = -1; > + > + switch (i) { > + case 0: > + num = scnprintf(str, size, "profile = %d\n", venc->params.profile); > + break; > + case 1: > + num = scnprintf(str, size, "level = %d\n", venc->params.level); > + break; > + case 2: > + num = scnprintf(str, size, "fps = %d/%d\n", > + venc->params.frame_rate.numerator, > + venc->params.frame_rate.denominator); > + break; > + case 3: > + num = scnprintf(str, size, "%d x %d -> %d x %d\n", > + venc->params.src_width, > + venc->params.src_height, > + venc->params.out_width, > + venc->params.out_height); > + break; > + case 4: > + num = scnprintf(str, size, "(%d, %d) %d x %d\n", > + venc->params.crop.left, > + venc->params.crop.top, > + venc->params.crop.width, > + venc->params.crop.height); > + break; > + case 5: > + num = scnprintf(str, size, > + "enable = 0x%x, input = %d, encode = %d, ready = %d, stopped = %d\n", > + venc->enable, > + venc->frame_count, venc->encode_count, > + venc->ready_count, > + venc->stopped); > + break; > + case 6: > + num = scnprintf(str, size, "gop = %d\n", venc->params.gop_length); > + break; > + case 7: > + num = scnprintf(str, size, "bframes = %d\n", venc->params.bframes); > + break; > + case 8: > + num = scnprintf(str, size, "rc: mode = %d, bitrate = %d, qp = %d\n", > + venc->params.rc_mode, > + venc->params.bitrate, > + venc->params.i_frame_qp); > + break; > + case 9: > + num = scnprintf(str, size, "sar: enable = %d, idc = %d, %d x %d\n", > + venc->params.sar.enable, > + venc->params.sar.idc, > + venc->params.sar.width, > + venc->params.sar.height); > + > + break; > + case 10: > + num = scnprintf(str, size, > + "colorspace: primaries = %d, transfer = %d, matrix = %d, full_range = %d\n", > + venc->params.color.primaries, > + venc->params.color.transfer, > + venc->params.color.matrix, > + venc->params.color.full_range); > + break; > + case 11: > + num = scnprintf(str, size, "skipped: count = %d, bytes = %d\n", > + venc->skipped_count, venc->skipped_bytes); > + break; > + default: > + break; > + } > + > + return num; > +} > + > +static struct vpu_inst_ops venc_inst_ops = { > + .ctrl_init = venc_ctrl_init, > + .check_ready = venc_check_ready, > + .input_done = venc_input_done, > + .get_one_frame = venc_frame_encoded, > + .buf_done = venc_buf_done, > + .stop_done = venc_stop_done, > + .event_notify = venc_event_notify, > + .release = venc_release, > + .cleanup = venc_cleanup, > + .start = venc_start_session, > + .mem_request = venc_request_mem_resource, > + .stop = venc_stop_session, > + .process_output = venc_process_output, > + .process_capture = venc_process_capture, > + .on_queue_empty = venc_on_queue_empty, > + .get_debug_info = venc_get_debug_info, > + .wait_prepare = vpu_inst_unlock, > + .wait_finish = vpu_inst_lock, > +}; > + > +static void venc_init(struct file *file) > +{ > + struct vpu_inst *inst = to_inst(file); > + struct venc_t *venc; > + struct v4l2_format f; > + struct v4l2_streamparm parm; > + > + venc = inst->priv; > + venc->params.qp_min = 1; > + venc->params.qp_max = 51; > + venc->params.qp_min_i = 1; > + venc->params.qp_max_i = 51; > + venc->params.bitrate_max = BITRATE_MAX; > + venc->params.bitrate_min = BITRATE_MIN; > + > + memset(&f, 0, sizeof(f)); > + f.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; > + f.fmt.pix_mp.pixelformat = V4L2_PIX_FMT_NV12M; > + f.fmt.pix_mp.width = 1280; > + f.fmt.pix_mp.height = 720; > + f.fmt.pix_mp.field = V4L2_FIELD_NONE; > + f.fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709; > + venc_s_fmt(file, &inst->fh, &f); > + > + memset(&f, 0, sizeof(f)); > + f.type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE; > + f.fmt.pix_mp.pixelformat = V4L2_PIX_FMT_H264; > + f.fmt.pix_mp.width = 1280; > + f.fmt.pix_mp.height = 720; > + f.fmt.pix_mp.field = V4L2_FIELD_NONE; > + venc_s_fmt(file, &inst->fh, &f); > + > + memset(&parm, 0, sizeof(parm)); > + parm.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE; > + parm.parm.capture.timeperframe.numerator = 1; > + parm.parm.capture.timeperframe.denominator = 30; > + venc_s_parm(file, &inst->fh, &parm); > +} > + > +static int venc_open(struct file *file) > +{ > + struct vpu_inst *inst; > + struct venc_t *venc; > + int ret; > + > + inst = vzalloc(sizeof(*inst)); > + if (!inst) > + return -ENOMEM; > + > + venc = vzalloc(sizeof(*venc)); > + if (!venc) { > + vfree(inst); > + return -ENOMEM; > + } > + > + inst->ops = &venc_inst_ops; > + inst->formats = venc_formats; > + inst->type = VPU_CORE_TYPE_ENC; > + inst->priv = venc; > + INIT_LIST_HEAD(&venc->frames); > + init_waitqueue_head(&venc->wq); > + > + ret = vpu_v4l2_open(file, inst); > + if (ret) > + return ret; > + > + venc_init(file); > + > + return 0; > +} > + > +static const struct v4l2_file_operations venc_fops = { > + .owner = THIS_MODULE, > + .open = venc_open, > + .release = vpu_v4l2_close, > + .unlocked_ioctl = video_ioctl2, > + .poll = v4l2_m2m_fop_poll, > + .mmap = v4l2_m2m_fop_mmap, > +}; > + > +const struct v4l2_ioctl_ops *venc_get_ioctl_ops(void) > +{ > + return &venc_ioctl_ops; > +} > + > +const struct v4l2_file_operations *venc_get_fops(void) > +{ > + return &venc_fops; > +}
> -----Original Message----- > From: Hans Verkuil [mailto:hverkuil-cisco@xs4all.nl] > Sent: Thursday, January 20, 2022 10:44 PM > To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org; > shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de > Cc: kernel@pengutronix.de; festevam@gmail.com; dl-linux-imx > <linux-imx@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>; > linux-media@vger.kernel.org; linux-kernel@vger.kernel.org; > devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org > Subject: [EXT] Re: [PATCH v14 00/13] amphion video decoder/encoder driver > > Caution: EXT Email > > Hi Ming, > > Running 'scripts/checkpatch.pl --strict' over the patches gives me lots of issues, > most of them easily fixed. > > As always with checkpatch, use common sense since not everything can be > fixed. > > You probably ran checkpatch.pl without the --strict option, but in the media > subsystem we use that option. > > Note that sparse and smatch are happy, so it is just the checkpatch issues that > need to be improved. Yes, I just run checkpatch.pl without the --strict option, I'll fix the issues in v15. > > On 12/7/21 09:24, Ming Qian wrote: > > Hi all, > > > > This patch series adds support for > > the amphion video encoder and decoder > > via the VPU block present in imx8q platforms. > > Currently, support for IMX8QXP and IMX8QM is included. > > > > It features decoding for the following formats: > > - H.264 > > - HEVC > > - MPEG4 > > - MPEG2 > > - VC1 > > - VP8 > > > > It features encoding for the following formats: > > - H.264 > > > > The driver creates a separate device node for the encoder and decoder. > > > > This driver is dependent on vpu firmwares. > > The firmwares have been submitted to linux-firmware. > > The firmware patch is since commit > > b563148fd28623f6b6ce68bb06c3dd3bd138b058: > > linux-firmware: Update firmware file for Intel Bluetooth 9462 (Fri Oct > > 8 16:30:14 2021 +0530) > > > > and it's available in the git repository at: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > > > ub.com%2Fmingqian-0%2Flinux-firmware.git&data=04%7C01%7Cming.qi > an% > > > 40nxp.com%7Ce806f02c9c204fab735408d9dc235f55%7C686ea1d3bc2b4c6fa > 92cd99 > > > c5c301635%7C0%7C1%7C637782866747675249%7CUnknown%7CTWFpbGZ > sb3d8eyJWIjo > > > iMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300 > 0& > > ;sdata=tGdn4wXog4281rkjx%2FkS5Yu995uf1dKk%2B%2Fw7g%2B4hiCc%3D > &rese > > rved=0 > > > > for you to fetch changes up to > bb3eee4f99589d4910dee4c053a3a685546b5dbb: > > amphion: add VPU firmwares for NXP i.MX8Q SoCs (Tue Oct 12 15:09:57 > > 2021 +0800) > > > > encoder is tested with gstreamer > > decoder is tested with gstreamer, but the following patches are required: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl > > > ab.freedesktop.org%2Fgstreamer%2Fgstreamer%2F-%2Fmerge_requests%2F1 > 379 > > > &data=04%7C01%7Cming.qian%40nxp.com%7Ce806f02c9c204fab73540 > 8d9dc23 > > > 5f55%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C63778286674 > 7675249%7 > > > CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT > iI6Ik1 > > > haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Z%2B28N2DXrRHm1PFwWrpk > 0koOSTVdqrm > > qqY6fKbJuh34%3D&reserved=0 > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitl > > > ab.freedesktop.org%2Fgstreamer%2Fgstreamer%2F-%2Fmerge_requests%2F1 > 381 > > > &data=04%7C01%7Cming.qian%40nxp.com%7Ce806f02c9c204fab73540 > 8d9dc23 > > > 5f55%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C63778286674 > 7675249%7 > > > CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT > iI6Ik1 > > > haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jitq4XO15%2BMux3jyVYGs0fp > Lwxr7hbv > > WAZad9TEGyso%3D&reserved=0 > > > > > > Tested-by: Nicolas Dufresne <nicolas.dufresne@collabora.com> > > For v15 please add this tag from Nicolas to the patch(es) that it applies to. > Tags in the cover letter are ignored.cd > > Regarding patch 12/13 (firmware patch): who will take that? I think it should > go through Shawn Guo, or if he can Ack it, then I can take it with the rest of > the series as well. > > Regards, > > Hans > > > > > > > Changelog: > > > > v14 > > - fix some errors according to Hans's comments > > > > v13 > > - make a workaround that avoid firmware entering wfi wrongly > > > > v12 > > - support reset decoder when starting a new stream > > - don't append an empty last buffer, set last_buffer_dequeued > > - improve the resolution change flow > > - return all buffers if start_streaming fail > > - fill encoder capture buffer's filed to none > > - fix a bug in calculating bytesperline > > > > v11 > > - fix dt_binding_check error after upgrade dtschema > > - remove "default y" > > - add media device > > > > v10 > > - refine vpu log, remove custom logging infrastructure > > - support non contiguous planes format nv12m instead of nv12 > > - rename V4L2_PIX_FMT_NV12_8L128 to V4L2_PIX_FMT_NV12MT_8L128 > > - rename V4L2_PIX_FMT_NV12_10BE_8L128 to > > V4L2_PIX_FMT_NV12MT_10BE_8L128 > > - merge two module into one > > - fix kernel panic in rmmod > > > > v9 > > - drop V4L2_BUF_FLAG_CODECCONFIG > > - drop V4L2_EVENT_CODEC_ERROR > > - drop V4L2_EVENT_SKIP - use the v4l2_buffer.sequence counter > > - fix some build warnings with W=1 reported by kernel test robot > > > > v8 > > - move driver from driver/media/platform/imx/vpu-8q to > > driver/media/platform/amphion > > - rename driver name to amphion > > - remove imx_vpu.h > > - move the definition of V4L2_EVENT_CODEC_ERROR to videodev2.h > > - move the definition of V4L2_EVENT_SKIP to videodev2.h > > > > v7 > > - fix build warnings with W=1 reported by kernel test robot > > > > v6: > > - rename V4L2_PIX_FMT_NT8 to V4L2_PIX_FMT_NV12_8L128 > > - rename V4L2_PIX_FMT_NT10 to V4L2_PIX_FMT_NV12_10BE_8L128 > > > > v5: > > - move some definition from imx_vph.h to videodev2.h > > - remove some unnecessary content > > - add some documentation descriptions > > - pass the lateset v4l2-compliance test > > > > v4: > > - redefine the memory-region in devicetree bindings documentation > > - use v4l2's mechanism to implement synchronize queuing ioctl > > - remove the unnecessary mutex ioctl_sync > > - don't notify source change event if the parameters are same as > > previously established > > - add flag V4L2_FMT_FLAG_DYN_RESOLUTION to decoder's capture format > > > > v3: > > - don't make vpu device node a simple-bus > > - trigger probing vpu core in the driver > > - remove unnecessary vpu core index property > > > > v2: > > - fix dt bindings build error > > - split driver patch into several parts to avoid exceeding bytes limit > > > > Compliance > > ========== > > # v4l2-compliance -d /dev/video0 > > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance > > SHA: 493af03f3c57 2021-10-08 17:23:11 > > > > Compliance test for amphion-vpu device /dev/video0: > > > > Driver Info: > > Driver name : amphion-vpu > > Card type : amphion vpu decoder > > Bus info : platform: amphion-vpu > > Driver version : 5.15.0 > > Capabilities : 0x84204000 > > Video Memory-to-Memory Multiplanar > > Streaming > > Extended Pix Format > > Device Capabilities > > Device Caps : 0x04204000 > > Video Memory-to-Memory Multiplanar > > Streaming > > Extended Pix Format > > Detected Stateful Decoder > > Media Driver Info: > > Driver name : amphion-vpu > > Model : amphion-vpu > > Serial : > > Bus info : platform: amphion-vpu > > Media version : 5.15.0 > > Hardware revision: 0x00000000 (0) > > Driver version : 5.15.0 > > Interface Info: > > ID : 0x0300000c > > Type : V4L Video > > Entity Info: > > ID : 0x00000001 (1) > > Name : amphion-vpu-decoder-source > > Function : V4L2 I/O > > Pad 0x01000002 : 0: Source > > Link 0x02000008: to remote pad 0x1000004 of entity > > 'amphion-vpu-decoder-proc' (Video Decoder): Data, Enabled, Immutable > > > > Required ioctls: > > test MC information (see 'Media Driver Info' above): OK > > test VIDIOC_QUERYCAP: OK > > test invalid ioctls: OK > > > > Allow for multiple opens: > > test second /dev/video0 open: OK > > test VIDIOC_QUERYCAP: OK > > test VIDIOC_G/S_PRIORITY: OK > > test for unlimited opens: OK > > > > Debug ioctls: > > test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) > > test VIDIOC_LOG_STATUS: OK (Not Supported) > > > > Input ioctls: > > test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) > > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > > test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) > > test VIDIOC_ENUMAUDIO: OK (Not Supported) > > test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) > > test VIDIOC_G/S_AUDIO: OK (Not Supported) > > Inputs: 0 Audio Inputs: 0 Tuners: 0 > > > > Output ioctls: > > test VIDIOC_G/S_MODULATOR: OK (Not Supported) > > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > > test VIDIOC_ENUMAUDOUT: OK (Not Supported) > > test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) > > test VIDIOC_G/S_AUDOUT: OK (Not Supported) > > Outputs: 0 Audio Outputs: 0 Modulators: 0 > > > > Input/Output configuration ioctls: > > test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) > > test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) > > test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) > > test VIDIOC_G/S_EDID: OK (Not Supported) > > > > Control ioctls: > > test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK > > test VIDIOC_QUERYCTRL: OK > > test VIDIOC_G/S_CTRL: OK > > test VIDIOC_G/S/TRY_EXT_CTRLS: OK > > test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK > > test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) > > Standard Controls: 3 Private Controls: 0 > > > > Format ioctls: > > test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK > > test VIDIOC_G/S_PARM: OK (Not Supported) > > test VIDIOC_G_FBUF: OK (Not Supported) > > test VIDIOC_G_FMT: OK > > test VIDIOC_TRY_FMT: OK > > test VIDIOC_S_FMT: OK > > test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) > > test Cropping: OK (Not Supported) > > test Composing: OK > > test Scaling: OK (Not Supported) > > > > Codec ioctls: > > test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) > > test VIDIOC_G_ENC_INDEX: OK (Not Supported) > > test VIDIOC_(TRY_)DECODER_CMD: OK > > > > Buffer ioctls: > > test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK > > test VIDIOC_EXPBUF: OK > > test Requests: OK (Not Supported) > > > > Total for amphion-vpu device /dev/video0: 46, Succeeded: 46, Failed: > > 0, Warnings: 0 > > > > # v4l2-compliance -d /dev/video1 > > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance > > SHA: 493af03f3c57 2021-10-08 17:23:11 > > > > Compliance test for amphion-vpu device /dev/video1: > > > > Driver Info: > > Driver name : amphion-vpu > > Card type : amphion vpu encoder > > Bus info : platform: amphion-vpu > > Driver version : 5.15.0 > > Capabilities : 0x84204000 > > Video Memory-to-Memory Multiplanar > > Streaming > > Extended Pix Format > > Device Capabilities > > Device Caps : 0x04204000 > > Video Memory-to-Memory Multiplanar > > Streaming > > Extended Pix Format > > Detected Stateful Encoder > > Media Driver Info: > > Driver name : amphion-vpu > > Model : amphion-vpu > > Serial : > > Bus info : platform: amphion-vpu > > Media version : 5.15.0 > > Hardware revision: 0x00000000 (0) > > Driver version : 5.15.0 > > Interface Info: > > ID : 0x0300001a > > Type : V4L Video > > Entity Info: > > ID : 0x0000000f (15) > > Name : amphion-vpu-encoder-source > > Function : V4L2 I/O > > Pad 0x01000010 : 0: Source > > Link 0x02000016: to remote pad 0x1000012 of entity > > 'amphion-vpu-encoder-proc' (Video Encoder): Data, Enabled, Immutable > > > > Required ioctls: > > test MC information (see 'Media Driver Info' above): OK > > test VIDIOC_QUERYCAP: OK > > test invalid ioctls: OK > > > > Allow for multiple opens: > > test second /dev/video1 open: OK > > test VIDIOC_QUERYCAP: OK > > test VIDIOC_G/S_PRIORITY: OK > > test for unlimited opens: OK > > > > Debug ioctls: > > test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) > > test VIDIOC_LOG_STATUS: OK (Not Supported) > > > > Input ioctls: > > test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) > > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > > test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) > > test VIDIOC_ENUMAUDIO: OK (Not Supported) > > test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) > > test VIDIOC_G/S_AUDIO: OK (Not Supported) > > Inputs: 0 Audio Inputs: 0 Tuners: 0 > > > > Output ioctls: > > test VIDIOC_G/S_MODULATOR: OK (Not Supported) > > test VIDIOC_G/S_FREQUENCY: OK (Not Supported) > > test VIDIOC_ENUMAUDOUT: OK (Not Supported) > > test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) > > test VIDIOC_G/S_AUDOUT: OK (Not Supported) > > Outputs: 0 Audio Outputs: 0 Modulators: 0 > > > > Input/Output configuration ioctls: > > test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) > > test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) > > test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) > > test VIDIOC_G/S_EDID: OK (Not Supported) > > > > Control ioctls: > > test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK > > test VIDIOC_QUERYCTRL: OK > > test VIDIOC_G/S_CTRL: OK > > test VIDIOC_G/S/TRY_EXT_CTRLS: OK > > test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK > > test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) > > Standard Controls: 20 Private Controls: 0 > > > > Format ioctls: > > test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK > > test VIDIOC_G/S_PARM: OK > > test VIDIOC_G_FBUF: OK (Not Supported) > > test VIDIOC_G_FMT: OK > > test VIDIOC_TRY_FMT: OK > > test VIDIOC_S_FMT: OK > > test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) > > test Cropping: OK > > test Composing: OK (Not Supported) > > test Scaling: OK (Not Supported) > > > > Codec ioctls: > > test VIDIOC_(TRY_)ENCODER_CMD: OK > > test VIDIOC_G_ENC_INDEX: OK (Not Supported) > > test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported) > > > > Buffer ioctls: > > test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK > > test VIDIOC_EXPBUF: OK > > test Requests: OK (Not Supported) > > > > Total for amphion-vpu device /dev/video1: 46, Succeeded: 46, Failed: > > 0, Warnings: 0 > > > > # v4l2-compliance -d /dev/media0 > > v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance > > SHA: 493af03f3c57 2021-10-08 17:23:11 > > > > Compliance test for amphion-vpu device /dev/media0: > > > > Media Driver Info: > > Driver name : amphion-vpu > > Model : amphion-vpu > > Serial : > > Bus info : platform: amphion-vpu > > Media version : 5.15.0 > > Hardware revision: 0x00000000 (0) > > Driver version : 5.15.0 > > > > Required ioctls: > > test MEDIA_IOC_DEVICE_INFO: OK > > test invalid ioctls: OK > > > > Allow for multiple opens: > > test second /dev/media0 open: OK > > test MEDIA_IOC_DEVICE_INFO: OK > > test for unlimited opens: OK > > > > Media Controller ioctls: > > test MEDIA_IOC_G_TOPOLOGY: OK > > Entities: 6 Interfaces: 2 Pads: 8 Links: 8 > > test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK > > test MEDIA_IOC_SETUP_LINK: OK > > > > Total for amphion-vpu device /dev/media0: 8, Succeeded: 8, Failed: 0, > > Warnings: 0 > > > > Ming Qian (13): > > dt-bindings: media: amphion: add amphion video codec bindings > > media: add nv12m_8l128 and nv12m_10be_8l128 video format. > > media: amphion: add amphion vpu device driver > > media: amphion: add vpu core driver > > media: amphion: implement vpu core communication based on mailbox > > media: amphion: add vpu v4l2 m2m support > > media: amphion: add v4l2 m2m vpu encoder stateful driver > > media: amphion: add v4l2 m2m vpu decoder stateful driver > > media: amphion: implement windsor encoder rpc interface > > media: amphion: implement malone decoder rpc interface > > ARM64: dts: freescale: imx8q: add imx vpu codec entries > > firmware: imx: scu-pd: imx8q: add vpu mu resources > > MAINTAINERS: add AMPHION VPU CODEC V4L2 driver entry > > > > .../bindings/media/amphion,vpu.yaml | 180 ++ > > .../media/v4l/pixfmt-yuv-planar.rst | 28 +- > > MAINTAINERS | 9 + > > .../arm64/boot/dts/freescale/imx8-ss-vpu.dtsi | 72 + > > arch/arm64/boot/dts/freescale/imx8qxp-mek.dts | 17 + > > arch/arm64/boot/dts/freescale/imx8qxp.dtsi | 24 + > > arch/arm64/configs/defconfig | 1 + > > drivers/firmware/imx/scu-pd.c | 4 + > > drivers/media/platform/Kconfig | 19 + > > drivers/media/platform/Makefile | 2 + > > drivers/media/platform/amphion/Makefile | 20 + > > drivers/media/platform/amphion/vdec.c | 1692 > +++++++++++++++++ > > drivers/media/platform/amphion/venc.c | 1354 +++++++++++++ > > drivers/media/platform/amphion/vpu.h | 356 ++++ > > drivers/media/platform/amphion/vpu_cmds.c | 437 +++++ > > drivers/media/platform/amphion/vpu_cmds.h | 25 + > > drivers/media/platform/amphion/vpu_codec.h | 67 + > > drivers/media/platform/amphion/vpu_color.c | 186 ++ > > drivers/media/platform/amphion/vpu_core.c | 872 +++++++++ > > drivers/media/platform/amphion/vpu_core.h | 15 + > > drivers/media/platform/amphion/vpu_dbg.c | 497 +++++ > > drivers/media/platform/amphion/vpu_defs.h | 186 ++ > > drivers/media/platform/amphion/vpu_drv.c | 260 +++ > > drivers/media/platform/amphion/vpu_helpers.c | 416 ++++ > > drivers/media/platform/amphion/vpu_helpers.h | 70 + > > drivers/media/platform/amphion/vpu_imx8q.c | 271 +++ > > drivers/media/platform/amphion/vpu_imx8q.h | 116 ++ > > drivers/media/platform/amphion/vpu_malone.c | 1630 > ++++++++++++++++ > > drivers/media/platform/amphion/vpu_malone.h | 42 + > > drivers/media/platform/amphion/vpu_mbox.c | 118 ++ > > drivers/media/platform/amphion/vpu_mbox.h | 16 + > > drivers/media/platform/amphion/vpu_msgs.c | 388 ++++ > > drivers/media/platform/amphion/vpu_msgs.h | 14 + > > drivers/media/platform/amphion/vpu_rpc.c | 265 +++ > > drivers/media/platform/amphion/vpu_rpc.h | 453 +++++ > > drivers/media/platform/amphion/vpu_v4l2.c | 733 +++++++ > > drivers/media/platform/amphion/vpu_v4l2.h | 56 + > > drivers/media/platform/amphion/vpu_windsor.c | 1207 ++++++++++++ > > drivers/media/platform/amphion/vpu_windsor.h | 39 + > > drivers/media/v4l2-core/v4l2-ioctl.c | 2 + > > include/uapi/linux/videodev2.h | 2 + > > 41 files changed, 12158 insertions(+), 3 deletions(-) create mode > > 100644 Documentation/devicetree/bindings/media/amphion,vpu.yaml > > create mode 100644 arch/arm64/boot/dts/freescale/imx8-ss-vpu.dtsi > > create mode 100644 drivers/media/platform/amphion/Makefile > > create mode 100644 drivers/media/platform/amphion/vdec.c > > create mode 100644 drivers/media/platform/amphion/venc.c > > create mode 100644 drivers/media/platform/amphion/vpu.h > > create mode 100644 drivers/media/platform/amphion/vpu_cmds.c > > create mode 100644 drivers/media/platform/amphion/vpu_cmds.h > > create mode 100644 drivers/media/platform/amphion/vpu_codec.h > > create mode 100644 drivers/media/platform/amphion/vpu_color.c > > create mode 100644 drivers/media/platform/amphion/vpu_core.c > > create mode 100644 drivers/media/platform/amphion/vpu_core.h > > create mode 100644 drivers/media/platform/amphion/vpu_dbg.c > > create mode 100644 drivers/media/platform/amphion/vpu_defs.h > > create mode 100644 drivers/media/platform/amphion/vpu_drv.c > > create mode 100644 drivers/media/platform/amphion/vpu_helpers.c > > create mode 100644 drivers/media/platform/amphion/vpu_helpers.h > > create mode 100644 drivers/media/platform/amphion/vpu_imx8q.c > > create mode 100644 drivers/media/platform/amphion/vpu_imx8q.h > > create mode 100644 drivers/media/platform/amphion/vpu_malone.c > > create mode 100644 drivers/media/platform/amphion/vpu_malone.h > > create mode 100644 drivers/media/platform/amphion/vpu_mbox.c > > create mode 100644 drivers/media/platform/amphion/vpu_mbox.h > > create mode 100644 drivers/media/platform/amphion/vpu_msgs.c > > create mode 100644 drivers/media/platform/amphion/vpu_msgs.h > > create mode 100644 drivers/media/platform/amphion/vpu_rpc.c > > create mode 100644 drivers/media/platform/amphion/vpu_rpc.h > > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.c > > create mode 100644 drivers/media/platform/amphion/vpu_v4l2.h > > create mode 100644 drivers/media/platform/amphion/vpu_windsor.c > > create mode 100644 drivers/media/platform/amphion/vpu_windsor.h > > > > > > base-commit: 1f1517fafda598839a02e39968c5063ddcfa51fc
Hi all, This patch series adds support for the amphion video encoder and decoder via the VPU block present in imx8q platforms. Currently, support for IMX8QXP and IMX8QM is included. It features decoding for the following formats: - H.264 - HEVC - MPEG4 - MPEG2 - VC1 - VP8 It features encoding for the following formats: - H.264 The driver creates a separate device node for the encoder and decoder. This driver is dependent on vpu firmwares. The firmwares have been submitted to linux-firmware. The firmware patch is since commit b563148fd28623f6b6ce68bb06c3dd3bd138b058: linux-firmware: Update firmware file for Intel Bluetooth 9462 (Fri Oct 8 16:30:14 2021 +0530) and it's available in the git repository at: https://github.com/mingqian-0/linux-firmware.git for you to fetch changes up to bb3eee4f99589d4910dee4c053a3a685546b5dbb: amphion: add VPU firmwares for NXP i.MX8Q SoCs (Tue Oct 12 15:09:57 2021 +0800) encoder is tested with gstreamer decoder is tested with gstreamer, but the following patches are required: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1379 https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 Tested-by: Nicolas Dufresne <nicolas.dufresne@collabora.com> Changelog: v14 - fix some errors according to Hans's comments v13 - make a workaround that avoid firmware entering wfi wrongly v12 - support reset decoder when starting a new stream - don't append an empty last buffer, set last_buffer_dequeued - improve the resolution change flow - return all buffers if start_streaming fail - fill encoder capture buffer's filed to none - fix a bug in calculating bytesperline v11 - fix dt_binding_check error after upgrade dtschema - remove "default y" - add media device v10 - refine vpu log, remove custom logging infrastructure - support non contiguous planes format nv12m instead of nv12 - rename V4L2_PIX_FMT_NV12_8L128 to V4L2_PIX_FMT_NV12MT_8L128 - rename V4L2_PIX_FMT_NV12_10BE_8L128 to V4L2_PIX_FMT_NV12MT_10BE_8L128 - merge two module into one - fix kernel panic in rmmod v9 - drop V4L2_BUF_FLAG_CODECCONFIG - drop V4L2_EVENT_CODEC_ERROR - drop V4L2_EVENT_SKIP - use the v4l2_buffer.sequence counter - fix some build warnings with W=1 reported by kernel test robot v8 - move driver from driver/media/platform/imx/vpu-8q to driver/media/platform/amphion - rename driver name to amphion - remove imx_vpu.h - move the definition of V4L2_EVENT_CODEC_ERROR to videodev2.h - move the definition of V4L2_EVENT_SKIP to videodev2.h v7 - fix build warnings with W=1 reported by kernel test robot v6: - rename V4L2_PIX_FMT_NT8 to V4L2_PIX_FMT_NV12_8L128 - rename V4L2_PIX_FMT_NT10 to V4L2_PIX_FMT_NV12_10BE_8L128 v5: - move some definition from imx_vph.h to videodev2.h - remove some unnecessary content - add some documentation descriptions - pass the lateset v4l2-compliance test v4: - redefine the memory-region in devicetree bindings documentation - use v4l2's mechanism to implement synchronize queuing ioctl - remove the unnecessary mutex ioctl_sync - don't notify source change event if the parameters are same as previously established - add flag V4L2_FMT_FLAG_DYN_RESOLUTION to decoder's capture format v3: - don't make vpu device node a simple-bus - trigger probing vpu core in the driver - remove unnecessary vpu core index property v2: - fix dt bindings build error - split driver patch into several parts to avoid exceeding bytes limit Compliance ========== # v4l2-compliance -d /dev/video0 v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 Compliance test for amphion-vpu device /dev/video0: Driver Info: Driver name : amphion-vpu Card type : amphion vpu decoder Bus info : platform: amphion-vpu Driver version : 5.15.0 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Detected Stateful Decoder Media Driver Info: Driver name : amphion-vpu Model : amphion-vpu Serial : Bus info : platform: amphion-vpu Media version : 5.15.0 Hardware revision: 0x00000000 (0) Driver version : 5.15.0 Interface Info: ID : 0x0300000c Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : amphion-vpu-decoder-source Function : V4L2 I/O Pad 0x01000002 : 0: Source Link 0x02000008: to remote pad 0x1000004 of entity 'amphion-vpu-decoder-proc' (Video Decoder): Data, Enabled, Immutable Required ioctls: test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK test invalid ioctls: OK Allow for multiple opens: test second /dev/video0 open: OK test VIDIOC_QUERYCAP: OK test VIDIOC_G/S_PRIORITY: OK test for unlimited opens: OK Debug ioctls: test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported) Input ioctls: test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported) test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported) Inputs: 0 Audio Inputs: 0 Tuners: 0 Output ioctls: test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported) Outputs: 0 Audio Outputs: 0 Modulators: 0 Input/Output configuration ioctls: test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported) Control ioctls: test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK test VIDIOC_G/S_CTRL: OK test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 3 Private Controls: 0 Format ioctls: test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK (Not Supported) test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK test VIDIOC_TRY_FMT: OK test VIDIOC_S_FMT: OK test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported) test Composing: OK test Scaling: OK (Not Supported) Codec ioctls: test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK Buffer ioctls: test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test VIDIOC_EXPBUF: OK test Requests: OK (Not Supported) Total for amphion-vpu device /dev/video0: 46, Succeeded: 46, Failed: 0, Warnings: 0 # v4l2-compliance -d /dev/video1 v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 Compliance test for amphion-vpu device /dev/video1: Driver Info: Driver name : amphion-vpu Card type : amphion vpu encoder Bus info : platform: amphion-vpu Driver version : 5.15.0 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Detected Stateful Encoder Media Driver Info: Driver name : amphion-vpu Model : amphion-vpu Serial : Bus info : platform: amphion-vpu Media version : 5.15.0 Hardware revision: 0x00000000 (0) Driver version : 5.15.0 Interface Info: ID : 0x0300001a Type : V4L Video Entity Info: ID : 0x0000000f (15) Name : amphion-vpu-encoder-source Function : V4L2 I/O Pad 0x01000010 : 0: Source Link 0x02000016: to remote pad 0x1000012 of entity 'amphion-vpu-encoder-proc' (Video Encoder): Data, Enabled, Immutable Required ioctls: test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK test invalid ioctls: OK Allow for multiple opens: test second /dev/video1 open: OK test VIDIOC_QUERYCAP: OK test VIDIOC_G/S_PRIORITY: OK test for unlimited opens: OK Debug ioctls: test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported) Input ioctls: test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported) test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported) Inputs: 0 Audio Inputs: 0 Tuners: 0 Output ioctls: test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported) Outputs: 0 Audio Outputs: 0 Modulators: 0 Input/Output configuration ioctls: test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported) Control ioctls: test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK test VIDIOC_G/S_CTRL: OK test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 20 Private Controls: 0 Format ioctls: test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK test VIDIOC_TRY_FMT: OK test VIDIOC_S_FMT: OK test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK test Composing: OK (Not Supported) test Scaling: OK (Not Supported) Codec ioctls: test VIDIOC_(TRY_)ENCODER_CMD: OK test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported) Buffer ioctls: test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test VIDIOC_EXPBUF: OK test Requests: OK (Not Supported) Total for amphion-vpu device /dev/video1: 46, Succeeded: 46, Failed: 0, Warnings: 0 # v4l2-compliance -d /dev/media0 v4l2-compliance 1.21.0-4859, 64 bits, 64-bit time_t v4l2-compliance SHA: 493af03f3c57 2021-10-08 17:23:11 Compliance test for amphion-vpu device /dev/media0: Media Driver Info: Driver name : amphion-vpu Model : amphion-vpu Serial : Bus info : platform: amphion-vpu Media version : 5.15.0 Hardware revision: 0x00000000 (0) Driver version : 5.15.0 Required ioctls: test MEDIA_IOC_DEVICE_INFO: OK test invalid ioctls: OK Allow for multiple opens: test second /dev/media0 open: OK test MEDIA_IOC_DEVICE_INFO: OK test for unlimited opens: OK Media Controller ioctls: test MEDIA_IOC_G_TOPOLOGY: OK Entities: 6 Interfaces: 2 Pads: 8 Links: 8 test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK test MEDIA_IOC_SETUP_LINK: OK Total for amphion-vpu device /dev/media0: 8, Succeeded: 8, Failed: 0, Warnings: 0 Ming Qian (13): dt-bindings: media: amphion: add amphion video codec bindings media: add nv12m_8l128 and nv12m_10be_8l128 video format. media: amphion: add amphion vpu device driver media: amphion: add vpu core driver media: amphion: implement vpu core communication based on mailbox media: amphion: add vpu v4l2 m2m support media: amphion: add v4l2 m2m vpu encoder stateful driver media: amphion: add v4l2 m2m vpu decoder stateful driver media: amphion: implement windsor encoder rpc interface media: amphion: implement malone decoder rpc interface ARM64: dts: freescale: imx8q: add imx vpu codec entries firmware: imx: scu-pd: imx8q: add vpu mu resources MAINTAINERS: add AMPHION VPU CODEC V4L2 driver entry .../bindings/media/amphion,vpu.yaml | 180 ++ .../media/v4l/pixfmt-yuv-planar.rst | 28 +- MAINTAINERS | 9 + .../arm64/boot/dts/freescale/imx8-ss-vpu.dtsi | 72 + arch/arm64/boot/dts/freescale/imx8qxp-mek.dts | 17 + arch/arm64/boot/dts/freescale/imx8qxp.dtsi | 24 + arch/arm64/configs/defconfig | 1 + drivers/firmware/imx/scu-pd.c | 4 + drivers/media/platform/Kconfig | 19 + drivers/media/platform/Makefile | 2 + drivers/media/platform/amphion/Makefile | 20 + drivers/media/platform/amphion/vdec.c | 1692 +++++++++++++++++ drivers/media/platform/amphion/venc.c | 1354 +++++++++++++ drivers/media/platform/amphion/vpu.h | 356 ++++ drivers/media/platform/amphion/vpu_cmds.c | 437 +++++ drivers/media/platform/amphion/vpu_cmds.h | 25 + drivers/media/platform/amphion/vpu_codec.h | 67 + drivers/media/platform/amphion/vpu_color.c | 186 ++ drivers/media/platform/amphion/vpu_core.c | 872 +++++++++ drivers/media/platform/amphion/vpu_core.h | 15 + drivers/media/platform/amphion/vpu_dbg.c | 497 +++++ drivers/media/platform/amphion/vpu_defs.h | 186 ++ drivers/media/platform/amphion/vpu_drv.c | 260 +++ drivers/media/platform/amphion/vpu_helpers.c | 416 ++++ drivers/media/platform/amphion/vpu_helpers.h | 70 + drivers/media/platform/amphion/vpu_imx8q.c | 271 +++ drivers/media/platform/amphion/vpu_imx8q.h | 116 ++ drivers/media/platform/amphion/vpu_malone.c | 1630 ++++++++++++++++ drivers/media/platform/amphion/vpu_malone.h | 42 + drivers/media/platform/amphion/vpu_mbox.c | 118 ++ drivers/media/platform/amphion/vpu_mbox.h | 16 + drivers/media/platform/amphion/vpu_msgs.c | 388 ++++ drivers/media/platform/amphion/vpu_msgs.h | 14 + drivers/media/platform/amphion/vpu_rpc.c | 265 +++ drivers/media/platform/amphion/vpu_rpc.h | 453 +++++ drivers/media/platform/amphion/vpu_v4l2.c | 733 +++++++ drivers/media/platform/amphion/vpu_v4l2.h | 56 + drivers/media/platform/amphion/vpu_windsor.c | 1207 ++++++++++++ drivers/media/platform/amphion/vpu_windsor.h | 39 + drivers/media/v4l2-core/v4l2-ioctl.c | 2 + include/uapi/linux/videodev2.h | 2 + 41 files changed, 12158 insertions(+), 3 deletions(-) create mode 100644 Documentation/devicetree/bindings/media/amphion,vpu.yaml create mode 100644 arch/arm64/boot/dts/freescale/imx8-ss-vpu.dtsi create mode 100644 drivers/media/platform/amphion/Makefile create mode 100644 drivers/media/platform/amphion/vdec.c create mode 100644 drivers/media/platform/amphion/venc.c create mode 100644 drivers/media/platform/amphion/vpu.h create mode 100644 drivers/media/platform/amphion/vpu_cmds.c create mode 100644 drivers/media/platform/amphion/vpu_cmds.h create mode 100644 drivers/media/platform/amphion/vpu_codec.h create mode 100644 drivers/media/platform/amphion/vpu_color.c create mode 100644 drivers/media/platform/amphion/vpu_core.c create mode 100644 drivers/media/platform/amphion/vpu_core.h create mode 100644 drivers/media/platform/amphion/vpu_dbg.c create mode 100644 drivers/media/platform/amphion/vpu_defs.h create mode 100644 drivers/media/platform/amphion/vpu_drv.c create mode 100644 drivers/media/platform/amphion/vpu_helpers.c create mode 100644 drivers/media/platform/amphion/vpu_helpers.h create mode 100644 drivers/media/platform/amphion/vpu_imx8q.c create mode 100644 drivers/media/platform/amphion/vpu_imx8q.h create mode 100644 drivers/media/platform/amphion/vpu_malone.c create mode 100644 drivers/media/platform/amphion/vpu_malone.h create mode 100644 drivers/media/platform/amphion/vpu_mbox.c create mode 100644 drivers/media/platform/amphion/vpu_mbox.h create mode 100644 drivers/media/platform/amphion/vpu_msgs.c create mode 100644 drivers/media/platform/amphion/vpu_msgs.h create mode 100644 drivers/media/platform/amphion/vpu_rpc.c create mode 100644 drivers/media/platform/amphion/vpu_rpc.h create mode 100644 drivers/media/platform/amphion/vpu_v4l2.c create mode 100644 drivers/media/platform/amphion/vpu_v4l2.h create mode 100644 drivers/media/platform/amphion/vpu_windsor.c create mode 100644 drivers/media/platform/amphion/vpu_windsor.h base-commit: 1f1517fafda598839a02e39968c5063ddcfa51fc