Building a synced video tryptic with Raspberry Pies and laptop screens
A filmmaker friend, Natasha Maidoff, approached me to design a display device that would allow her to display a video triptych at the 2015 Venice Art Walk. The idea was to create a frame that would contain all the components necessary to play three different videos on three separate screens in perfect sync. It should be simple and should not need to be connected to a bunch of cables or computers. The goal was to hang it, plug it, turn it on and forget it… Oh, and did I mention it should not cost an arm and a leg?
Based on my previous tinkering activities, I felt that this project was within my reach and so I started researching parts. My first order was for 3 Raspberry Pi 2 mini computers, 3 replacement laptop LCD screens and the 3 driver boards they needed to be connected to the Pi’s HDMI output. Along with that, I got the various necessary AC adapters to power all this mess.
Framing and mounting the electronics
Networking the three Raspberry Pi
For this to work reliably and predictably, I changed the Pies’ IP address to be static, and while I was there, I changed the hostnames, and set it up to automatically log in to the command line without loading the desktop interface.
To set up the static IP to 192.168.11.45, edit /etc/network/interfaces and replace:
iface wlan0 inet dhcp
with:
address 192.168.11.45
netmask 255.255.255.0
gateway 192.168.11.1
network 192.168.11.1
broadcast 192.168.11.255
To setup the hostname, replace it to your desired name in /etc/hostname, and also list all the other hostnames with their static IPs in /etc/hosts.
Lastly, to log directly into the command line interface without having to enter a username and password, edit the file /etc/inittab and replace the line:
1:2345:respawn:/sbin/getty 115200 tty1
with:
1:2345:respawn:/bin/login -f pi tty1 /dev/tty1 2>&1
Getting rid of warts
Wire time… AC power cord into 12V power supply, 3 sets of wires with DC power barrel jacks connecting the 12V output to each of the monitors, 1 set of wires connecting to the 12V to 5V step down converter. 5V output of the down converter to the raspberry pies and the ethernet hub. One of the challenges here was trying to solder wire to the micro USB plugs I had bought. It’s all way too small and I ended up with an ugly heap, kind f like Jeff Goldblum and the fly in “The Fly”, except with melted lead and plastic. In the end, I just bought some USB cables with the proper connection, cut them and rewired them to fit my needs.
The software
Auto start
I needed the movies to start automatically when the machines were done booting, so I modified the /home/pi/.bashrc file to run my main script whenever it was being accessed at the end of a boot sequence (I didn’t want to launch my movie playback script every time I logged in from another machine). I look for the $L1 environment variable that gets to “tty1” only when logging to the console from the boot sequence. I added the following at the top of .bashrc:
sudo /home/pi/video_player.py
fi
Movie playback
For movie playback, I made the obvious choice and used omxplayer which is custom written for the Raspberry Pi hardware and can play full 30fps HD video from the GPU, and there’s even a crude little library called pyomxplayer that allows control from python. In order to get the pyomxplayer library to run, I had to install the pexpect python library which allows it script to spawn and control the omxplayer process. Also, pyomxplayer tries to parse the text output by omxplayer but it seems like that part of the code has changed and causes the script to fail and exit so I had to remove that part of the code. I also added a function to allow me to rewind the movie. As soon as my script starts, omxplayer loads the appropriate movie file and pauses at the beginning.
Syncing the movies
As for syncing the start of the three movies, I used pyOSC to have the machines automatically establish a connection when they boot up and unpause the movies at the same instant when all three machines are ready. The basic process goes like this: I designate one machine to be the master and the two others to be slaves. When the master boots up, it first listens for a signal from each the slaves, and stays in this mode until it has heard from both. On their end, the slaves’ first action during launch is to send a signal to the master. As soon as the master has heard from both slaves, it tells the slaves to switch to a state where they listen to the master for commands. At this point, the master unpauses the movie it previously loaded and tells the slaves to do the same with theirs. Since omxplayer has no looping function I could find that worked for me, I have the master wait for the length of the movie and then rewind to movies to the beginning and start them playing over again.
Playing back from the RAM disk
In order to avoid continually reading from the SD card, I created a ram disk so my script could copy the movie file to it and allow omxplayer to play back from there.
I created the directory for the ram disk’s mount point:
sudo mkdir /var/ramdisk
and added the following to the /etc/fstab file:
ramdisk /var/ramdisk tmpfs nodev,nosuid,size=500M 0 0
The Raspberry Pi 2 comes with 1 GB of ram so I used half of it for the drive, which left plenty for the OS to run.
Timing issues
Fundamentally, pyomxplayer and the pexpect approach it uses is an ingenious but somewhat hacky way to control the process and it took me a long time to get everything to work properly. I found that if my script sent commands to omxplayer too fast, omxplayer would miss the command. I had to put a bunch of sleep statements in my code to pause and allow enough time for commands to get properly “heard” by the omxplayer process. Also, I added a long pause right after the movie is first loaded into omxplayer to make sure any residual and proc intensive boot process has a chance to finish and does not interfere with the responsiveness of each machine. It’s far from a robust setup; It’s basically a script that tells three machines to press the “play” button at the same time. Ultimately, though, I seemed to be able to get the movies to sync well enough across all three machines. Not guaranteed to be frame perfect every time but probably within one or two frames on a reliable basis. Ultimately, I would love for someone to write a true python API for omxplayer.
Powering off and rebooting through GPIO
Since the frame will be fully autonomous, it will not have a keyboard to allow an operator to properly power off the Raspberry Pies. This is a challenge because if you directly unplug the power to turn them off, the SD card will most likely get corrupted and you will not be able to reboot the machine. So, I needed to setup a simple way to cleanly shutdown the machines before turning the power off. I also wanted to be able to reboot if somehow something unexpected happened that threw the machine out of sync. So, I connected the master machine to a couple of switches and set the script up to power off or reboot if the GPIO interface detects a specific button press.
The code
Here is the code. You will need to install pyomxplayer, pexpect, and pyOSC. Also, depending on your version of omxplayer, you may need to modify pyomxplayer because of the way it attempts to parse omxplayer’s output. This script works for both the master and the slave based on the hostname. Here is the basic process:
When all machines boot up, they copy the relevant media to the ram drive, and launch it in omxplayer in a paused state
The slaves then go into a startup loop sending a “ready” address to the master until they hear back.
The master goes into a loop to check if both slaves are ready, listening for a “ready” address from each.
When the master determines that both slaves are ready, it sends a “listen” address to the slaves and the slaves come out of their startup loop.
From here on out, the master controls the slaves through OSC to unpause/pause/rewind the movies indefinitely.
The machines either reboot or shut down when the specific GPIO pin gets set on the master.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 |
#!/usr/bin/python import OSC import threading, socket, shutil from pyomxplayer import OMXPlayer from time import sleep ''' ''' hosts = {'master':'192.168.11.45', 'slave1':'192.168.11.46', 'slave2':'192.168.11.47'} movies = {'master':'/home/pi/media/movie_lf.mov', 'slave1':'/home/pi/media/movie_cn.mov', 'slave2':'/home/pi/media/movie_rt.mov'} movieLength = 60*5 hostname = socket.gethostname() print "copying %s to /var/ramdisk" % movies[hostname] shutil.copy(movies[hostname], "/var/ramdisk/") movies[hostname] = movies[hostname].replace('/home/pi/media', '/var/ramdisk') print "playing movie from %s" % movies[hostname] omx = OMXPlayer(movies[hostname]) sleep(5) omx.toggle_pause() sleep(1) omx.rewind() sleep(1) def reboot(): command = "/usr/bin/sudo /sbin/shutdown -r now" import subprocess process = subprocess.Popen(command.split(), stdout=subprocess.PIPE) output = process.communicate()[0] print output def poweroff(): command = "/usr/bin/sudo /sbin/shutdown -h now" import subprocess process = subprocess.Popen(command.split(), stdout=subprocess.PIPE) output = process.communicate()[0] print output if hostname == 'master': def gpio_check(): import RPi.GPIO as GPIO GPIO.setmode(GPIO.BCM) GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_UP) GPIO.setup(24, GPIO.IN, pull_up_down=GPIO.PUD_UP) while True: input_state = GPIO.input(23) if input_state == False: print('Detected a reboot request: Button 23 Pressed') send_reboot(slaves['slave1'][2]) send_reboot(slaves['slave2'][2]) sleep(.3) reboot() input_state = GPIO.input(24) if input_state == False: print('Detected a poweroff request: Button 24 Pressed') send_poweroff(slaves['slave1'][2]) send_poweroff(slaves['slave2'][2]) print('Button 24 Pressed') sleep(.3) poweroff() sleep(.1) gpio_thread = threading.Thread(target=gpio_check) gpio_thread.start() # send listen def send_listen(slaveServer): print "querying slave", slaveServer msg = OSC.OSCMessage() msg.setAddress("/listen") # set OSC address msg.append(" the master.\nThe master is pausing for 20 seconds.") # int slaveServer.send(msg) # send it! # send play command def send_play(slaveServer): print "sending play to", slaveServer msg = OSC.OSCMessage() msg.setAddress("/play") # set OSC address msg.append("the master") # int slaveServer.send(msg) # send it! # send toggle_pause command def send_toggle_pause(slaveServer): print "sending toggle_pause to", slaveServer msg = OSC.OSCMessage() msg.setAddress("/toggle_pause") # set OSC address msg.append("the master") # int slaveServer.send(msg) # send it! # send rewind command def send_rewind(slaveServer): print "sending rewind to", slaveServer msg = OSC.OSCMessage() msg.setAddress("/rewind") # set OSC address msg.append("the master") # int slaveServer.send(msg) # send it! # send reboot command def send_reboot(slaveServer): print "sending reboot to", slaveServer msg = OSC.OSCMessage() msg.setAddress("/reboot") # set OSC address msg.append("the master") # int slaveServer.send(msg) # send it! # send poweroff command def send_poweroff(slaveServer): print "sending poweroff to", slaveServer msg = OSC.OSCMessage() msg.setAddress("/poweroff") # set OSC address msg.append("the master") # int slaveServer.send(msg) # send it! # handler for ready address def ready_handler(addr, tags, stuff, source): if not slaves[stuff[0]][0]: slaves[stuff[0]][0] = True print "setting %s to ready" % stuff[0] # setup clients to send messages to slavesReady = False c1 = OSC.OSCClient() c2 = OSC.OSCClient() slaves = {'slave1':[False, (hosts['slave1'], 9000), c1] , 'slave2':[False, (hosts['slave2'], 9000), c2] } # set up self to receive messages receive_address = hosts['master'], 9000 s = OSC.OSCServer(receive_address) # basic s.addDefaultHandlers() s.addMsgHandler("/ready", ready_handler) # adding our function # Start OSCServer print "\nStarting OSCServer. Use ctrl-C to quit." st = threading.Thread( target = s.serve_forever ) st.start() # set up clients to send messages to slaves['slave1'][2].connect( slaves['slave1'][1] ) # set the address for all following messages slaves['slave2'][2].connect( slaves['slave2'][1] ) # set the address for all following messages ######### # establish communication print "Master is waiting to hear from the slaves." # The master waits until both slaves are ready while not slavesReady: sleep(.01) if slaves['slave1'][0] and slaves['slave2'][0]: slavesReady = True print "The master has heard from both slaves" # The master tells the slaves to listen send_listen(slaves['slave1'][2]) send_listen(slaves['slave2'][2]) print "The master has told the slaves to listen" print "Pausing for 20 seconds" # catch our breath sleep(20) ######### # media control # we go into an infinite loop where we # unpause, wait for the movie length # pause, wait, rewind, wait, unpause print "entering main loop" while True: send_toggle_pause(slaves['slave1'][2]) send_toggle_pause(slaves['slave2'][2]) omx.toggle_pause() sleep(movieLength) send_toggle_pause(slaves['slave1'][2]) send_toggle_pause(slaves['slave2'][2]) omx.toggle_pause() sleep(2) send_rewind(slaves['slave1'][2]) send_rewind(slaves['slave2'][2]) omx.rewind() sleep(2) else: thisName = hostname thisIP = hosts[hostname], 9000 masterStatus = {'awake':[False], 'play':[False]} masterAddress = hosts['master'], 9000 def send_ready(c): msg = OSC.OSCMessage() msg.setAddress("/ready") # set OSC address msg.append(thisName) # int try: c.send(msg) except: pass def listen_handler(add, tags, stuff, source): print "I was told to listen by%s" % stuff[0] masterStatus['awake'][0] = True def play_handler(add, tags, stuff, source): print "I was told to play by %s" % stuff[0] masterStatus['play'][0] = True def toggle_pause_handler(add, tags, stuff, source): print "I was told to toggle_pause by %s" % stuff[0] omx.toggle_pause() def rewind_handler(add, tags, stuff, source): print "I was told to rewind by %s" % stuff[0] omx.rewind() masterStatus['awake']=False def reboot_handler(add, tags, stuff, source): print "I was told to reboot by %s" % stuff[0] reboot() def poweroff_handler(add, tags, stuff, source): print "I was told to poweroff by %s" % stuff[0] poweroff() ########### # create a client to send messages to master c = OSC.OSCClient() c.connect( masterAddress ) ########### # listen to messages from master receive_address = thisIP s = OSC.OSCServer(receive_address) # basic # define handlers s.addDefaultHandlers() s.addMsgHandler("/listen", listen_handler) s.addMsgHandler("/play", play_handler) s.addMsgHandler("/toggle_pause", toggle_pause_handler) s.addMsgHandler("/rewind", rewind_handler) s.addMsgHandler("/reboot", reboot_handler) s.addMsgHandler("/poweroff", poweroff_handler) # Start OSCServer print "\nStarting OSCServer. Use ctrl-C to quit." st = threading.Thread( target = s.serve_forever ) st.start() print "%s connecting to master." % hostname while True: #########i## # keep sending ready signals until master sends a message # on the /listen address which gets us out of this loop while not masterStatus['awake'][0]: sleep(.01) send_ready(c) ########## # once the master has taken control, we do nothing # and let the master drive playback through handlers |
Framing
The last few details consisted of mounting the tactile switches connected to the GPIO pins for shutting down or rebooting the machines, soldering on a mini jack connected to the master Raspberry Pi and provide audio from the outside, and lastly, installing a main power switch for the whole unit.
Hanging
Here it is… We’re very happy! The work looks great. After seeing Natasha’s piece running on it in perfect sync, we’re all very inspired to work with this tryptic format. There are so many ideas to explore. Since we’re hoping for this to sell at the auction for the Venice Family Clinic, we’re already planning to build another one for us to play with.
Parts list
In case you are foolish enough to go down the same route I did, here’s what I used:
3 raspberry pi 2
3 LTN156AT02-D02 Samsung 15.6″ WXGA replacement screens (ebay)
3 HDMI+DVI+VGA+Audio Controller Board Driver Kit for LTN156AT02 (ebay)
1 D-Link – 5-Port 10/100 Mbps Fast Ethernet Switch
DC 12V 10A Regulated Transformer Power Supply (typically used for LED strips)
DC-DC 12V/ 24V to 5V 10A Converter Step Down Regulator Module
Aluminum, mini usb cables, 3 ethernet cables, 3 ethernet cables, nuts and bolts, tactile switch, power switch, wire…