VJing Projection Mapping Mac & PC live coding VDMX Madmapper.

 

I thought it would be a good idea to document the process used for Cosmic Sound’s Altared event w/  Bary Center, J Butler, To Sleep At Night, Dilettante. It wasn’t my first time using a multiple projector/computer setup, but adding a PC and Live coding  VJ into the mix needed a new workflow. There’s a video from the show below and if you’re interested in the process of sharing live VJing content across computers and operating systems read the entire post.

Here are some short video clips from Altared I co-VJ’ed with the awesome Char Stiles.

GLSL, Live coding VJing, Mac & PC, projection mapping, VDMX, Mad Mapper, Kodelife, Lumen App, Spout, Syphon, UVC HDMI Capture, four channel HDMI mixers, and fun!

That statement above says it all. Now I’ll go into the details.

As Char writes on her blog:  ( http://charstiles.com/performing-coding/   ). We were using a Windows 10 Laptop running Madmapper and Kodelife. While there is little documentation on Kodelife, it is very intuitive if you understand the world of GLSL and live creative coding.  Simply put, GLSL is a shader language that runs directly on the GPU. Skip the 4-hour blender or After effects rendering queue and process visuals in real time. Change a few lines of code and tweak them as you go.  This style of VJing offers some big live performance pros and cons. And as time goes on, it will only get better.

 

Whether you know it or not, a good number of VJ and projection mapping applications have been making use of GLSL shaders for years.  VDMX, for instance, has their brand of shader called and ISF. Modul8, Madmapper, Resolume, etc. The pro’s of GLSL is in the performance possibilities.  Here’s a youtube video that rendered out in real time and its executable file size is tiny in comparison to the capture video size. The downside is you can change a line of code and end up with a VJ blank screen of death (VBSOD).

Char and I wanted to collaborate, and what better way then for both of us to share our VJ feeds with one another.  I get her’s, and she’ll have mine. Allowing both of us to do our own thing or tweak each other’s feed. This process led to some visceral results but took a few trials runs to get it all figured out.  Our setup looked something like this:

Using a combination of NDI, Syphon, Spout, and a USB 3.0 HDMI capture device we could share our feeds, loop them, and mix/blend/mash all of the above.

 

Here was one hurdle we had to tackle.  Kodelife -> to Spout -> to NDI. Kodelife works with Spout (PC) & Syphon (Mac) natively which is immensely helpful for sharing its real-time graphics output on the same machine.  But when you want to share it with another computer, two options are an Ethernet cable or an HDMI capture box. (I’ve written about these boxes here: link ) For this, we did both, but either or would have worked.  The problem was that the Spout to NDI .exe application needs reset when you open up another Kodelife shader ( or scene ). My MacBook Pro with VDMX & Madmapper could easily import the feed from Char’s PC running Kodelife, but every time the scene would change, the NDI connection would get dropped.  This is where the magic of Madmapper came in. Madmapper imports spout & Syphon natively. So when Kodelife is running, madmapper see’s Kodelife’s Spout or Syphon output. Madmapper also has NDI send and receiver built-in. And even better, 1 Madmapper license is good for up to two machines. By using madmapper -> NDI  over ethernet to my Mac also running Madmapper, we were able to workaround the issue of the Spout to NDI windows program reset. Madmapper sends a signal as long as the application is open.

 

So Char’s Kodelife -> Spout -> Madmapper -> NDI -> over an ethernet cable -> to my Macbook pro running Madmapper -> VDMX -> back to my Madmapper -> out to a projector, was a huge chunk of our setup.   While this may seem congested, there was little to no latency from this process. From laptop to laptop you can use a single CAT6 ( CAT5 will work ) to share visuals back and forth. We also used my Roland 4 input HDMI mixer as a hardware device that could select either of our feeds and override the output to the projector.  So if Char wanted to run the show, she could slide a lever and take control. Vice versa, but this also gives us the ability to work with more VJ’s or bring in live camera feeds, feedback loops, etc. (Enter -> Altared II on 3.31.2018 🙂 Char, ProjectileObjects, and Nate Cake will all be VJing together using this setup. By adding a router or switch into the mix, Nate will be able to share his NDI output from Modul8, the VJ software that he likes to use.

 

It’s a blessing to be able to collaborate with such remarkable individuals.  Thankfully the technology continues to evolve to make that more possible. Thanks to Char Stiles, Hexler.net <- the makers of Kodelife. Madmapper, VDMX, and all the individuals who made this whole process possible.

 

projectileobjects led ring slowdanger

SlowDanger LED Ring Moving Shadows

I’ve known the slowdanger duo for some years now. During a recent residency with them and Bill Shannon. We had time to talk about a future collab. I sent them this video of moving shadows. (gif now).

They liked the idea and wanted it to be portable for touring. I tossed around some ideas but a series of spinning lights seemed impractical. I had some RGB LED strips laying around and after a few night experiments that are too embarrassing to post, I built a giant ring to test.  The 50′ circumference ring breaks down into 10x 5′ segments with quick connectors.  The whole kit can fit into the trunk of a car and can run off a Raspberry Pi and pixel controller.

The first day I got it up and running, I was almost trampled over by the office dog.

Our original launch date was sometime in the spring, but a show came up at the Wood Street galleries, and they asked if I could slap it together in time. It’s still a work in progress, but one that is fun to play with. Similar to the theater, it looks even better in person and hard to capture on camera. Take a peek; we hope to do more with it soon!

Best way to capture and share VJ content?

What’s the best way to capture VJing live?

I put this article together to share some of the best ways I’ve found over the years to capture an HDMI signal from a computer.  Additionally, it also covers ways to share visuals from one computer to another so that two or more VJ’s can work together.

This article is broken up into two sections. Standalone capture boxes and UVC devices that require a computer.  If you’re only interested in sharing VJ content from one computer to the next, skip to UVC devices.

If you’re familiar with HDCP, screen resolutions, and Syphon/Spout; feel free to skip ahead. I won’t go into the details about color encoding /spacing or codec compression, but I’ve included links at the bottom of this article.

HDCP and the pain it can cause  

Most computers, MAC’s notoriously, will sometimes require an HDCP handshake before they send a video signal. This digital copyright protection protocol is implemented to keep you from bootlegging content.   Unfortunately, even if you’re sending out a VJ signal you own, there is no ‘legal’ way to disable HDCP within the software or hardware of your computer;  But, you can bypass it…

A VGA cable is the most “cost-effective” way to dodge HDMI, but a VGA capture device can be harder and more expensive to find. :/

An HDMI splitter is another possible way, but not all are the same – – And later you may find out that some playback devices won’t work at all if they do not detect HDCP. I don’t know too many VJ’s using an Apple TV or Amazon Fire TV stick, but it can happen. For this, there’s better ways than splitting or converting the signal.

My first foray into the world of capture devices took me on a collision course with HDCP. Nothing I tried would work, and the only thing that worked was expensive and problematic. In 2012, I landed on a Blackmagic HyperDeck Shuttle SSD recorder. It had a battery, captured HDMI or SDI, and didn’t have a problem with my MacBook pro’s HDCP connection. I hit record, and…  the uncompressed files were MASSIVE, the SSD drives were expensive, and only a few SSD chipsets were compatible. VJing on my laptop was enough to cook an egg, so using built-in software capture was out of the question. I needed all the GPU and CPU power that I could get.

 

“What I was look for?”           

HDMI in & out, HD recording 1080p30 or higher, and low to no latency. I tested dozens of “streaming boxes/capture devices” that would add as much as a 2 second delay. For me VJing is about timing and synesthesia and a 2-second delay is not going to work…

Standalone Capture devices:

vj recorder box capture device VJing Projectileobjects

All the devices from Cloner-Alliance work and spoof HDCP. If you do a quick google search for HDMI USB capture, you will find boxes that look almost identical to cloner alliance’s, and guess what, they are. I’ve even tested cloner alliance’s firmware updates on the Evolve look-alike box that I bought off Ebay. (Update firmware at your own risk, I cannot promise that all internal hardware will be compatible). I will say avoid their Flint LX box at all costs. It has a 720p image processor inside that adds a sharpening filter to cook out a 1080p signal. It is not a real 1080p image and even adds a terrible amount of latency.

Most of these boxes are simple, insert a thumb drive (or TF card = microSD), plug in your HDMI source and connect it to a projector or monitor. Press a single button to start and stop recording. It should take a few seconds to clean up the .mp4 file, but that’s it. You now have an H.264 video ready for sharing, streaming, or playback.* ( H.264 .MP4 is not an optimized codec for live visuals, look into HAP if your VJ software supports it or DXV for Resolume.)

Avermedia Portable gamer will not work with HDCP signals. (Sorry Kevin for not warning you sooner.) THANKS to a comment from That Fuzzy Bastard after posting this article. He uses Thunderbolt to DVI, then puts a Cable Unlimited DVI to HDMI (AUD-2362) box between the computer and the Avermedia. (see in comments).

 

UVC and Thunderbolt Devices:

magewell capture usb blackmagic thunderbolt capture vj mutliple vjing capture device

These devices require a computer to capture or “share” the content from one VJ to the next. Most free up CPU power by handling the image processing on board. Intern, your computer treats it like a USB 3.0 webcam and pulls in the signal with almost no latency. There are PCIe versions of this, but I VJ with laptops. A majority of UVC devices are designed for streaming or compression. Some aim to produce an optimized video streaming or compress video for smaller file sizes. In doing this, it takes more time for the onboard processors and adds video latency (or delay). In my tests, there were only three devices that met my needs for low latency. The best by far was a MAGEWELL USB capture HDMI Plus which can handle a 2k signal. MAGEWELL seems to offer the highest quality for UVC capture devices but at a $$$ more expensive price point. ($359 at the time of this post). The best overall for price and performance that I found is the iEi HDB-301R-R10. ($149.99), despite the iEi specifications saying that is only works with 10.13+, it worked perfectly on OSX 10.12 and Windows 10 without additional drivers. The cheapest is a generic capture card that someone else has already reviewed in greater detail.

I’m still waiting to test cloner alliance Flint LXT, MOKOSE USB3.0 HDMI, and MOKOSE HDMI.

The Elgato game capture HD60 S was a terrible disappointment. It has a small form factor, competitive price, and low latency, but it doesn’t work with anything outside of Elgato’s proprietary software center. Likewise; these didn’t work: https://www.amazon.com/gp/product/B00PC5HUA6/ nor this https://www.amazon.com/gp/product/B06XWL7SZD/

🙂 Thunderbolt:

I have a love-hate relationship with Thunderbolt, most of it hate, but the low price point of the BlackMagic Ultrastudio mini recorder was pretty hard to beat $149 (+ price of Thunderbolt cable), and it was able to transport an HD video signal from one computer to the next with low latency. It requires Thunderbolt 1 or 2, which can be limiting for PC users. Designed for Broadcast and TV standard resolutions, not computer resolutions! The max resolution the BMD mini recorder supports is 1920x1080i60 ← You may have seen something like 1080p or 1280x720p60, or 59.94i? The “i” is interlaced, and if there’s a “p,” then it’s progressive.

Think of the “p” as a full picture so that p60 would be 60 pictures per second, p30 = 30. TV’s are changing, but they have long since run at 29.97fps or 59.94 interlaced fps. Here’s a helpful photo.

interlaced vs progressive vj capture devices

If there’s latency or dropped frames, the interlaced lines can become visible over the top of your footage. You can remove them with a post process, but not so easily during a live performance. (one of those things YOU as a VJ would notice, that the rest of the audience might not).\n

To avoid this, you’ll have to send a progressive signal into the Mini Recorder. It supports 720p50, 720p59.94, 720p60, 1080p23.98, 1080p24, 1080p25, 1080p29.97, 1080p30. I use SwitchResX to force mac computer outputs to my desired resolution. Another downside is that the mini recorder does not support 16:10 aspect ratios; This forces me to crop or stretch the resolution when I’m working on a WUXGA projector. Simply put, if you’re on a Mac, the Black Magic mini recorder is really good, but far from perfect. (It supports some Windows machines with Thunderbolt). And as of posting this article. I sold my mini recorder and purchased a magewell. But something new or better will come out soon, or maybe it is already here (NDI?)

Bonus sectionSyphon / Spout, TCP, NDI, etc.:

Syphon (Mac) and Spout (PC) are graphics sharing pipelines that can share video across applications on the same computer, and with TCPSyphon or NDI, you can send that signal over a network (Gigabit recommended). These are fast and usually, do not require additional hardware (unless your on a stupid MacBook pro that needs thirty USB-C dongles for you to VJ). NDI is excellent if you’ve got your setup working. I like the UVC USB 3.0 devices because they are compact and BUS powered, but if you can haul an ethernet connection to your next event, then maybe NDI is your next best friend.

http://spout.zeal.co/download-spout-to-ndi/

For Resolume users:

https://resolume.com/support/en/syphonspout

 

WHAT ABOUT MULTIPLE VJ’S???

I am the proud owner of a Roland V-1HD 4 input HDMI mixer.  It has BPM sync, midi in & out and it’s compact.

 

I hope this article helped. If you found any better devices or found this to be useful, please leave a comment or message. Thanks -Cornelius

 

Additional Resources:

Display Resolutions:

https://en.wikipedia.org/wiki/Display_resolution#/media/File:Vector_Video_Standards8.svg

 

Color spacing:

https://en.wikipedia.org/wiki/Chroma_subsampling

http://blogs.adobe.com/VideoRoad/2010/06/color_subsampling_or_what_is_4.html

http://www.clearone.com/blog/what-is-444-color-and-why-should-i-care/

https://en.wikipedia.org/wiki/Color_space

 

Encoding:

https://en.wikipedia.org/wiki/YUV

https://en.wikipedia.org/wiki/RGB_color_model

https://en.wikipedia.org/wiki/YCbCr

https://en.wikipedia.org/wiki/YPbPr

http://paulbourke.net/dataformats/nv12/

 

Codec:

https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

https://en.wikipedia.org/wiki/Motion_JPEG

 

Catcalling is Harassment Stickers #catcallingisharassment

20170624_093259

Project update:   Due to the paper stickers lack of weatherproofing (indoor only) I have pivoted into having a more “permanent” sticker made.  Introducing the #CatcallingIsHarassment eggshell stickers.  They adhere to smooth clean surfaces and become difficult to remove after about 5 minutes.  They are now stationed at a few local coffee shops in the Pittsburgh area. Thus far we made 12,500 stickers.

You can download the design in the previous post (below) to make your own.

20170622_111828

—– Previous post.

This isn’t in the realm of projection mapping, but I’m sure some night soon this will be brightly displayed on the side of a building.   For now I’m using this blog as a way to share the idea.

catcallingisharassment catcalling is harassment stickers

#CatcallingIsHarassment stickers

It only took $163.14 to make 3,000 4″x4″ stickers that 86% of them “disappeared” in less than a day.   I have since updated the design to incorporate the hashtag #CatCallingIsHarassment to help others share the idea and uploaded the files for anyone to download, share, or printing.  i.e. signs & stickers.

Catcalling Is Harassment Files

Catcalling is Harassment PDF only

Initially someone printed 500 Vinyl stickers for free, they were amazing quality, but limited, more expensive, easier to remove (in one piece), and more detrimental to the environment. For paper stickers I found this company had good prices and turn around time, I’m going to see what else is out there, but for now: https://www.printplace.com/products/promotional-and-event-stickers

Hopefully you can take it from here, spread the word, et cetera. Let me know if you find a better printer.

#catcallingisharassment #catcalling

EspressoAMano Projection mapping pittsburgh, projectileobjects, coffee shop window projection

Update and EspressoAMano open projection mapping project.

 I’ve been contributing to projection-mapping.org more frequently than my own blog and it has been some time since I last posted an update. For me, it’s a good problem to have, I’ve been busy…

Over the last 2 years I’ve been piling up a collection of unedited videos from a slew of projects that I’ve had the honor to be apart of.  What were once planned to be written detailed accounts for educational purposes, may well end up becoming a flashy promo reel propelled by high energy EDM pop music.  <- Or not.

It’s what I have seen everywhere else and while I understand it, it’s not in my immediate interests to create one, or to make bids on larger corporate events.  While I may consult  on such events from time to time, believe it or not, I’m in this for the fun of it.

A few days ago I finished an update for a local automated projection mapping installation.  The goal, take a cool local coffee shop and light up the upper windows at night. The ulterior motive for this? To see what others can do with the visual space when given the opportunity.

 EspressoAMano Projection mapping pittsburgh, projectileobjects, coffee shop window projection

Share the link, download the template, submit your work and it will be viewed publicly.  That’s it. https://www.evernote.com/l/Ag2xwxA4QRtJE4z8E8lZ7x8jbwDmkIZXrvM

 

Only time will tell what’s going to happen next with this installation, but the opportunity is something that I am excited about.  Check back in a few weeks or so as I will have a detailed update of how I made this installation possible and the simple “built-in” means of projection automation.

 

The location:

Espresso A Mano (https://espressoamano.com/)

3623 Butler St, Pittsburgh, PA 15201

Current Projector Operating hours (subject to change):

Mon-Thur:

9PM-11PM

Fri & Sat:

9PM-2AM

Sunday:

6PM-9PM

 

 

OMG 5 Tunnel Party

OMG 5 Tunnel Party Uploading to Youtube vs Facebook

Its been a while since I’ve a moment to blog and even longer since the first OMG parties, including our infamous U-Haul Dance Party (<-Video / Blog Post <-).  So we did it again, but this time we had a tunnel dance party. <- Why not?.  The video was quickly put together as a recap and shared on Youtube & Facebook. Interestingly enough, Facebook favors its own video content so heavily that within 24 hours of posting the Tunnel Party video we reached more than 5,000 people compared to our youtube page which only reached 5.   5000 is a 99900% increase of 5.  So why even upload to youtube?  Archival, SEO, and Quality. While facebook is the social sharing hub of our time, videos disappear just as fast as they arrive. They share with friends, and friends of friends, but don’t rank highly in search engine results. Lastly, to optimize streaming across all devices, facebook really dumbs down the quality of any video uploaded to it.  That being said, I hope you enjoy our tunnel party video as much as we enjoyed our tunnel party!.  Thank you to everyone who came out.

Raspberry Pi Zero projection mapping

Raspberry pi zero projection mapping

Update:  If the content is pre-mapped (meaning, made in After Effects, Photoshop, MS Paint, etc.) and rendered out at the projector’s native resolution.  The Pi Zero works great as a standalone seamless video looper by simply embedding this code into the rc.local file.  (I stored a video file on a usb flash drive and entitled it “Video.mp4”  You can also put your video file on the MicroSD that Raspbian is installed on.

omxplayer -b -o --no-osd --loop /mnt/usb/Video.mp4

Old —–

Built off of the 1st gen. versions of the raspberry pi, the Pi Zero is a $5 computer that seems fit for simple projection mapping and playback.  Thus far it’s limitations have simple fixes, but a lack of ethernet and additional usb ports is hoop that has to be jumped through. (My current setup is micro usb to 4-port usb 2.0 hub with nano wifi dongle & Logitech k400).

OMG 3, Pittsburgh

The OMG Experiments – UHaul Dance Party

***Video at the bottom if you’d like to skip***

Its been a while since I’ve had the time to blog, my entire year has been non-stop and before I could post my latest works I was already off to do something new. All in all, it has been a trip that I am thankful to be apart of.  More recently I decided to experiment with MadMapper 2.0, A Pixlite 16 LED driver, and 25 meters of WS2811 RGB strips.  Oh yeah, and all inside of a mobile Uhaul party vehicle powered by a 2000 watt silent generator…  Let’s just say this was a spectacular experiment.

For some time now I have been doing some fun pop-up / experimental dance parties with my neighbor and friend Joy ( DJ name “Kill Joy” <- I think).  We just like to try new things and that is how OMG started.  Original, “Only Mashups Goobers” due to the first event being at a location that did not pay BMI or ASCAP, so we had a mashup party and devised a visual setup using 1 projector, cardboard, duck-tape, spray paint, and some hot pants.  This is what the first OMG party looked like:

But even crazier than the actual party was the media campaign that was released on Facebook two weeks before. Quick edited, simply narrated, “hot-topic” videos with blown out audio that looked and sounded spectacular on mobile devices (most of facebooks’ traffic) were a hit amongst those who saw them. Here are some of the OMG videos:

Not bad, and they had tons of shares.  So that was OMG 1, after that was OMG 2, The Sledgehammer dance party….   Let’s just say this party was so crazy it’s almost like it never even happened.

And then, there was OMG 3, The Mobile UHaul LED Disco Dance Party…