These are the commands I ran to replace the docker-compose that ships with the Docker package on DSM.

Update docker-compose

sudo su
cd /var/packages/Docker/target/usr/bin/
mv docker-compose docker-compose_bak
curl -L https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-`uname -s`-`uname -m` -o docker-compose
chmod +x ./docker-compose

Jellyfin with hardware transcoding

This is the compose file I use to run Jellyfin on the NAS.

version: '3.5'
services:
  jellyfin:
    image: jellyfin/jellyfin
    container_name: jellyfin
    user: 1026:100
    group_add:
      - "937"
    network_mode: 'host'
    volumes:
      - /volume2/docker/jellyfin/config:/config
      - /volume2/docker/jellyfin/cache:/cache
      - /volume2/Media/:/media
    restart: 'unless-stopped'
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
    # Optional - may be necessary for docker healthcheck to pass if running in host network mode
    extra_hosts:
      - "host.docker.internal:host-gateway"

To get group id of renderD128 device:

cat /etc/group | grep videodriver | cut -d: -f3

This will allow hardware accelerated encoding.

Further reading

To utilize/get more value out of Synology: Docker on Synology DSM

Related: Self-hosting ArchiveBox with Docker Compose on Synology.