Python vs Emacs Lisp Text Processing

April 28, 2018

Post-production For asciinema Screencast Files

In the JSONL screencast file below, created with asciinema, a little editing will make for a much smoother screencast.

There is a 12 second pause in the beginning, start typing the command cat Vagrantfile, but I change my mind and remove what I’ve typed, and type head -n 50 Vagrantfile instead. After seeing the output, I end the screencast by typing exit.

I want to remove all of the typing, except the head command, but then the screencast will start with a 16 second pause. To fix this, I need to re-number all of the timecode values after removing the appropriate lines.

{"timestamp": 1524245288, "height": 50, "env": {"TERM": "xterm-256color", "SHELL": "/bin/bash"}, "version": 2, "width": 160}
[0.199148, "o", "vagrant@inky vagrant$ "]
[12.302205, "o", "vagrant@inky vagrant$ "]
[14.603106, "o", "c"]
[14.691388, "o", "a"]
[14.77987, "o", "t"]
[14.852587, "o", " "]
[15.155492, "o", "V"]
[15.390311, "o", "agrantfile "]
[15.660129, "o", "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b"]
[15.828665, "o", "\u001b[K"]
[16.043649, "o", "h"]
[16.108335, "o", "e"]
[16.148271, "o", "a"]
[16.243907, "o", "d"]
[16.588573, "o", " "]
[16.804049, "o", "-"]
[17.070766, "o", "n"]
[17.212697, "o", " "]
[17.532887, "o", "5"]
[17.637986, "o", "0"]
[17.764699, "o", " "]
[18.037386, "o", "V"]
[18.244977, "o", "a"]
[18.463647, "o", "grantfile "]
[18.741398, "o", "\r\n"]
[18.746395, "o", "# -*- mode: ruby -*-\r\n\r\nVagrant.configure(\"2\") do |config|\r\n
[18.747524, "o", "vagrant@inky vagrant$ "]
[20.430896, "o", "e"]
[20.590255, "o", "x"]
[20.718599, "o", "i"]
[20.806351, "o", "t"]
[20.910807, "o", "\r\nexit\r\n"]

Simple Approach and Sophisticated Approach

The simple approach (the one I’ll use right now) is to set the first timecode to 0 and then re-number in a nice increment like 100 milliseconds.

A more sophisticated approach is to use thresholds, which would clean things up while maintaining a more human feel. In other words, short pauses and long pauses would all get cut to 100 milliseconds, which would make all typing fast and crisp, and would remove any long sections of dead air. But pauses in a range of maybe half a second to five seconds would remain as they are, so looking at output and considering what to do next would feel natural.

The Python Solution

import json

OUTPUT_FILENAME = 'orignal.cast'
INPUT_FILENAME = 'processed.cast'

screencast = []
timecode = 0.0

with open(INPUT_FILENAME, 'r') as fh:
    for line in fh:
        screenshot_data = json.loads(line)
        if isinstance(screenshot_data, list):
            screenshot_data[0] = timecode
        timecode += 0.1

with open(OUTPUT_FILENAME, 'w') as fh:

The point of the Python code is to work with the semantics of the input file (a newline delimted list of JSON values) and take advantage of Python’s built-in datatypes and standard library. I could treat the input as raw text and write some less expensive code, but then why bother with Python?

One improvement I’d like to make is to avoid building up the entire output in memory in a Python list. It will probably never be a problem, but I have to hope that the input file is always a reasonable size. Python automatically streams the input, so it would be nicer to operate the same way on the output. However, I’m interested in looking at how I’d code up a solution quickly, and the above code is what I’d do if I’m just trying to get something working and move on.

The Emacs Lisp Solution

(defvar fixtime-timecode (float 0))

(defun fixtime-increment-timecode ()
  (setq fixtime-timecode (+ fixtime-timecode 0.1)))

(defun fixtime-replace-next-timecode ()
  (let (start end)
    (setq start (point))
    (search-forward ",")
    (setq end (point))
    (delete-region start end)
    (insert (number-to-string fixtime-timecode))))

(defun fixtime-find-timecode ()
  (search-forward-regexp "^\\[" nil t))

(defun fixtime ()
  (while (fixtime-find-timecode)

While the Python code works with semantics of the input file, the Elisp code follows how I’d process the text manually. I believe I could write something in Emacs that is closer to what I wrote in Python, especially if I use a library to parse JSON, but this is definitely how I’d initially approach the problem with Emacs. I’m looking at the input file on my screen and running the code in my editor, so it makes sense to think about the solution in terms of scripting my editor.

Which One Will I Use?

I like that the Python solution is operating on a more meaningful level, as opposed to the Emacs Lisp code which operates on a “character” level. I like how straightforward and readable the Elisp code is.

To use the Python code I have to go to the terminal and I need to coordinate the Python file, an input file, and the output file. The big win for the Emacs code, obviously, is that I can just run it in Emacs.

The really cool thing is that I’m not too far from using the simple Emacs Lisp approach in a way that mimics what I want from the sophisticated approach. The code already runs from where the cursor is in the buffer, which means I can skip over lines that I don’t want to re-number. If I make the code work on a region of selected text, then I’m on my way to creating an asciinema post-processing application inside of Emacs.

In Python I create a program that processes the text, but in Emacs I extend my ability to work on the text interactively.

Regular Expressions

The Emacs Lisp code translates pretty directly into a regular expression. I could use a regular expression solution in either Python or Emacs, but that’s a dumb way to compare Python vs Emacs. Regular expressions are their own language.

In Emacs it would look something like this:

(defun fixtime-with-re ()
  "Find and replace using regex."
  (re-search-forward "^\\[\\([0-9\\.]+\\)")
  (replace-match (number-to-string fixtime-timecode) nil nil nil 1))

Tags: programming emacs