Close

comskip for docker emby

wjcarpenterWJCarpenter wrote 04/27/2024 at 22:33 • 6 min read • Like

This is a description of how I have set up commercial skipping during playback of my emby recordings.

I have a few ways of recording live/linear TV. I have both good antenna OTA reception for local channels and various (legal/paid) accesses to other channels. The two server applications I use most for playing or recording live TV are Channels DVR and emby. I run both of them in docker containers on the same Raspberry Pi 4.

There is a popular application called comskip that analyzes video files and uses several tunable heuristics to detect commercial advertisement breaks. It can output the list of breaks in any of several standard or quasi-standard formats. Probably the most popular output format is EDL, edit decision list. Commercial skipping is a two-part process. First, the video file is analyzed (which to some extent can be done while it's being played or recorded). Second, when the video is played, the player understands the results of the analysis and provides for either manual or automatic skipping of commercial breaks.

Channels DVR has support for comskip built in, both for the analysis and playback. It's easily configured via the web GUI. emby does not have comskip built in, but it can be added via the sophisticated Emby.Comskipper plugin. The plugin provides the playback skipping based on EDL files. (It also recognizes some specially designated advertising chapter markers sometimes present in video files, though that has nothing to do with comskip.) The author provides detailed and complete instructions for how to hook in the comskip analysis piece to the emby server.

Although I run emby in a docker container, I wanted to run comskip either in its own container or on the host system. I didn't find a convenient comskip docker image, so I used the glue described below to run comskip on the host RPi 4. Running comskip with 3 threads, the CPU cores are loaded at about 75% each during actual analysis. For now, I only run a single comskip job at a time.

comskip-inotify.sh

This is a short shell script that manages the kick-off of comskip jobs. Jobs can be triggered in one of two ways.

  1. When the script starts, it looks for video files in the emby recording area that do not have a companion EDL file. This is a decent enough catch-up technique for crashes, reboots, etc.
  2. After that, it waits for the arrival of new video files from emby. That's done via the inotify-wait tool watching for creation events in the entire emby recording area.

For either job type, a job is created using task spooler. It's a straightforward queuing tool available in most Linux package repositories.

There are a couple of odds and ends to deal with in the script.

Without further ado, here is comskip-inotify.sh:

#!/bin/bash

# Top level directory under which emby puts the Live TV recordings (default can be overridden on the command line)
RECORDINGS=${1:-/mnt/red-drive/emby/programdata/data/livetv/recordings/}

consider_one_file() {
    DIRECTORY=${1}
    FILE=${2}
    HOWCOME=${3}
    case ${HOWCOME} in
	FIND)
	    # only process files with a .ts extension where there is not already an EDL file
	    if [ "${FILE}" != "${FILE%.ts}" -a ! -e "${DIRECTORY}${FILE%.ts}.edl" ]
	    then
		queue_one_comskip ${HOWCOME} "${DIRECTORY}" "${FILE}"
	    fi
	    ;;
	WAIT)
	    # only process files with a .ts-READY extension
	    if [ "${FILE}" != "${FILE%.ts-READY}" ]
	    then
		rm -f "${DIRECTORY}${FILE}"
		queue_one_comskip ${HOWCOME} "${DIRECTORY}" "${FILE%-READY}"
	    fi
	    ;;
    esac
}

queue_one_comskip() {
    HOWCOME=${1}
    DIRECTORY=${2}
    FILE=${3}
    FULLPATH="${DIRECTORY}${FILE%-READY}"

    echo "${HOWCOME} $(date) \"${FULLPATH}\""
    # Need a full path for comskip because tsp doesn't automatically spawn a shell to run the queued command.
    # We don't need to include the file in the label since it shows up in the full command line of the queued item.
    tsp -L "comskip ${HOWCOME}" /usr/local/bin/comskip --ini=/etc/comskip.ini "${FULLPATH}"
    if [ ${HOWCOME} == "WAIT" ]
    then
	# Make processing of new files take priority over catch-up files:
	tsp -u
    fi
    # Neither of these are perfect, but it's just for de-clutter.
    # clear out records for finished jobs every once in a while
    tsp -C
    # task spooler accumulates the stdout files in /tmp/. I tried using "-n", but for some reason that
    # caused the first comskip job to never terminate and move on to the next jobs.
    rm -f /tmp/ts-out.*
}

# This is to catch up with any unprocessed files. In the steady state, should be few and far between.
find ${RECORDINGS} -type f -name '*.ts' |
    while read DF
    do
	DIRECTORY=$(dirname "${DF}")
	FILE=$(basename "${DF}")
        consider_one_file "${DIRECTORY}/" "${FILE}" FIND
    done    

# Now, wait for new recordings to show up.
# Use format to put each item on a separate line because spaces and commas want to drive us crazy.
inotifywait -q -m -r -e create ${RECORDINGS} -s --format "%w%n%e%n%f" |
    while true;
    do
	read DIRECTORY;
	read EVENT;
	read FILE;
	# strict "CREATE" since directories use "CREATE,ISDIR"
	if [ "${EVENT}" == "CREATE" ]
	then
            consider_one_file "${DIRECTORY}" "${FILE}" WAIT
	fi
    done

/etc/comskip.ini

I'm still tuning the comskip config items. There are a lot. I started with the file prepared and recommended by the author of the emby plugin, and then I modified a few things based on values chosen by Channels DVR, which has been pretty decent at detection. Channels DVR sets only a dozen or so config values and takes the defaults for the rest. Most notably, I turned off the "live_tv" setting because I don't use that, and I configured comskip to also emit a closed caption file in SRT format. The CC thing is due to an unrelated bug in some of the emby clients that fails to find closed captions in a subset of my recordings.

emby postprocessing

I don't want to start the comskip job until the recording is completed. There are comskip ways to analyze files while they are being recorded, but they involve some compromises that I don't want. There's not a completely reliable way for inotify-wait to detect that emby is finished with a recording. Instead, we use emby postprocessing to create a signal file. It's just an empty file with the name of the recording suffixed with "-READY". The comskip-inotify.sh script is waiting for the appearance of those files instead of the recordings themselves.

emby postprocessing is configured in the server settings: Live TV > Advanced > Recording Post Processing. We use this simple command:

systemd details

To run comskip-inotify at system boot time, or to manually control it during troubleshooting, I use the following /etc/systemd/system/comskip-inotify.service service definition.

[Unit]
Description=Commercial skip processing of emby recordings
After=network.target

[Service]
ExecStart=/shares/comskip-inotify.sh /mnt/red-drive/emby/programdata/data/livetv/recordings/
Restart=always
User=dockerruntime
Group=dockerruntime
Type=simple

[Install]
WantedBy=multi-user.target

 Of course, on someone else's system, the location of the comskip-inotify.sh script and the root of the emby live recording tree would be in different locations.

permissions dancing

On my host system, I define a pseudo user "dockerruntime" and a group with the same name. I configure the emby docker container to run with that user and group. You can see in the service definition above that I run the service under the same user and group. The reason for having the close agreement is so that the emby GUI will be properly able to delete things, including the comskip EDL and SRT files.

Via some defaulting mechanism, my emby container was originally running as "bin:bin". That would have been OK for the service definition, but I wanted to be able to check the task spooler queue. That didn't work with the locked down "bin" user. I check it now with "su dockerruntime -c tsp".

Like

Discussions