Project Brief:

Create a Web based VR museum to showcase the creative coding that went on in Carrie's class in the past 5 weeks. Incorporate Arduino's and custom microcontrollers as an optional way of interaction within the VR Museum. House the re-imagined digital exhibition of Mark Rothko as a separate wing of the VR Museum.

‍

Materials:

• Arduino Nano IoT 33
• Wires x N/A
• Pushbuttons x 4
• Breadboard
• Computer

‍

Design Process:

I've worked with A-Frame before but it's definitely been a while. I opted to model out the entire museum form with smaller box geometries. I chose to use smaller box geometries mainly because it was easier to texture. Since I wasn't creating the entire museum in a different 3D software and texturing according to UV mapping, any texture that I used with A-Frame would quickly become distorted or stretched dramatically. I debated going into a 3D modeling software for the museum walls as I had seen in https://cecropia.github.io/thehallaframe/ a project that I was greatly inspired by. With Cecropia however I realized that the scope of their museum was much smaller than mine and more of a proof of concept / experimentation with WebVR performance. As I knew that there would be set number of pieces that I needed to show off, my museum design very much was informed by what would be included inside.

‍

I originally pictured an octagonal design where users would be able to walk around the relatively circular shape and observe the different pieces. As this would be a museum designed specifically to display these pieces instead of a white box where art is always coming and going I wanted to eschew the typical boxy white wall designs found in many fine art galleries.

‍

A large part of the ideation came from the https://www.npmjs.com/package/aframe-websurfaces Web surface A-Frame component, I erroneously believed that this component would allow me to create a super interactive museum where users would be able to access the sketches themselves on layers of P5 web editor. This turned out not the be case as it required very machine heavy calculations to run especially with the mouse pointer raycasting, also due to the way three.js works and the underlying webGL works it's actually not possible to project into VR space. Or at least at the moment it's not really a possibility to calculate fast enough for VR usage.

‍

After creating 4 octagonal rooms I took a step back and realized that in terms of design, the museum was relatively bland and also somewhat repetitive. At first I attempted to create a more interesting visual by adding dynamic lights to each of the rooms, originally I intended to have a spot light over each piece to really create dramatic environment. My FPS then proceeded to drop from 60 to 7 and after some digging around I found that point lights and lights in general, especially those that cast dynamic shadows absolutely kill your machine. The only option to have so many dynamic lights would be to take the objects into a 3D software and pre-bake all the lights and shadows in. Given the time constraints/my own inexperience with 3D modeling and software, would not be a real option.

I eventually settled on a mix of primitive built and pre-made 3D models that I found for free online. Using a stylized modern chandelier light I found on CGtrader and a single larger spot light I was able to create a more dynamic 'fuller' look to the octagonal rooms at minimal cpu/gpu overhead.

The coding part was really quite simple there isn't much calculating for me to do or being clever, really it's just placing legos in place and making sure they don't clip...which actually is more busy work and time consuming than it was difficult. For some reason in A-Frame for certain geometries if you place them perfectly back to back they just start clipping through each other like crazy? It is not reproducible phenomena however as it doesn't always happen? I have yet to figure out exactly when and why it does that but at times it does and I literally have to make micro adjustments down to 0.001 at a time to make sure no visible gaps are seen and also that it doesn't clip. A very annoying trial and error process, it was also annoying to get the name plates under everyone correctly, because for whatever reasoning the A-text component is left aligned in the weirdest way possible. If you try and use math and put them all 5.5 away from each other the same way the sketches were positioned they will all be off by varying amounts. Another eye ball and trial and error process.

I eventually did some re-designing and created a more typical gallery designed room for the final room which would house the final presentations. When out numbered 3-1 the more typical boxy design actually stands on its own quite well and contrasts the previous rooms and helps signify the difference in art pieces displayed inside.

The back of the final project room opens into a long corridor that I rigged to fade into blackness wherein at the very end of the tunnel, audio will play and users will get to see my re-imagined Rothkos. With the way lighting works in 3D and Three.js in particular I decided to work smarter and not harder and instead of customizing lighting effects to fade into black, I would draw custom texture that was of the original wall fading into black.

‍

The last part of the project was to work with the Arduino and the custom gamepad. I had a custom camera already in the A-frame sketch since I was utilizing https://github.com/wmurphyrd/aframe-physics-extras, Will Murphy's physics library and the custom Kinema Camera. The Kinema camera worked with the default WASD control component so I knew it was a matter of using node and socket to create serial communication and spoofing 'wasd' or up down left right commands via the Arduino.

Of course it wouldn't be me if I didn't take apart my setup to begin prototyping my picture book project and then subsequently fry the living hell out of all my equipment. I'm currently unsure of whether it's the breadboard that's donezo or the Arduino but I can't even light up a single LED at this current moment.

‍

Demo:

Live Site:

https://bho.media/SummerWebVRC/

https://github.com/fakebrianho/webVRCreativeCodingMuseum

‍

‍

Learning Experiences:

1. Read through the entire documentation you fool, don't base your ideation on a single concept or connection before testing it.
2. Cannibalize your own work, turns out 2.5 years ago when I last touched A-Frame I was much better than I am now, my documentation was waaaaay better than some current documentation.
3. Work smarter not harder, use math you fool instead of eye balling things, it's better to take time to figure things out and then apply smartly than brute force implementation.
4. You're probably never going to like your project or feel like you've done a good job, keep ignore that voice and keep doing things.

‍

Troubleshooting: Issues I ran into / Unresolved

1. Websurfaces just don't work in VR, confirmed by lead A-Frame dev RIP. The WebGL brute force windowing also has been patched out by three.js updates.
2. Raycasting is difficult to implement and also costly, custom camera made it so typical cursor element no long sticks.
3. Geometry clipping seemingly randomly, not reproducible, had to make custom 0.0001 adjustments for certain elements.
4. Look controls + movement controls together cause lag spikes and frame buffering issues leading to unintentional teleporting and clipping.

Future Progression:

I'm relatively happy with what I managed to build in a single week, the next steps would involve heavy dev ops and optimization. Modeling and pre-baking lighting for the museum frame. Perhaps extending museum into bigger space or using TP to create multiple pages representing different floors or even playing with literal verticality with physics engine.

‍

For the Arduino and controllers it would definitely be moving into Wifi websocket connections or exploring commercial/industrial level wifi cards to get past the latency issues that I've had in the past with wifi-movement controls. Then a 3D printed casing and buttons etc would be nice for housing purposes.

‍

Creating second page for Ellen's class, adding buttons for custom controller to switch in-between the two webpages.