by arisAlexis ·
Not many people read books but AI is under rated for the ability to theme your experience. If you ask Gemini to create a youtube music playlist for you for a specific book the result is amazing and I wanted to share this with others because it changed my book reading to an actual audio visual experience.
There are some apps that do that but the songs are not on point, too "theme like". Gemini can find real songs that are a super fit.
My original idea was to make a modern supercharged version of SAM, the old-school text-to-speech. Then I remembered Pink Trombone, one of my favorite creative coding projects. In the end, I decided to just make it a model that mimics speech - it would be easy to make a text-to-speech by chaining an existing TTS with a Samuel filter.
The phonetics work quite well for vowels - you can try making a sound and then seeing if the predicted tongue prediction matches where your tongue was. It's less accurate for consonants; it doesn't do plosives or nasals (I tried).
How the machine learning works:
It's a speech autoencoder in which the decoder is Pink Trombone. A couple of issues: PT is not differentiable, so I use Gumbel softmax to get a differentiable approximation. PT also generates audio sample-by-sample (apparently it's called a "Kelly-Lochbaum vocal tract model"), which would make training extremely slow. To do that, I approximate it using FIR filters that change each 10ms. The fundamental frequency is not computed by the model, but by a method called pyin. Now that the gradient can flow through the model, the question is what loss to use. Matching the spectrum/MFCC coefficients gets you to something that sounds like speech, but isn't intelligible. What does work is trying to match the original and generated audio in the space of a self-supervised audio representation model - I used wav2vec2. There are a couple of tricks on top of that like encouraging the model not to move the parameters too quickly, and balancing the Gumbel softmax between exploration and exploitation.
I’m a Technical Artist. I built anyCreature, an open-source, mesh-based generative harness designed to programmatically quantify and enforce 3D aesthetics through three automated pipeline checks: 1. QC Thresholds: Automated pass/fail checks for standard game-engine specs (poly count, vertex budget, rigging, and animation tracks). 2. Quantified Aesthetics: Spatial vertex analysis to objectively measure mass distribution and form tension, ensuring biologically logical anatomy. 3. Silhouette & Readability: Automated high-contrast B&W renders and 24px thumbnail tests to guarantee visual clarity and instant readability in-game.
I would love to see what you create with this tool. Feedback is welcome as I continue to optimize this harness.
Ariescar | Gobkit