What could be better than an artificial intelligence capable of converting your scribbles into a real masterpiece? A Replicate engineer, known by the GitHub username zeke, created just that. Scribble Diffusion is an open-source application that turns your handmade drawings into a work of art.
Scribble Diffusion is an open-source application that turns your hand-drawn drawings, along with a text description, into a work of art. It’s not the first nor the last tool to turn scribbles into drawings. AutoDraw could already turn paths into drawings, as explained Numerama.
Based on ControlNet
The tool is based on ControlNet, a neural network framework developed by Lvmin Zhang and Maneesh Agrawala. The model combines ControlNet with Stable Diffusion. It thus uses your scribbles and text as packaging to drive the image generation.
ControlNet picks up your squiggles and detects its outlines using the Canny edge finder. Other tools are used, such as Openpose which is a quick keypoint detection model. The latter can extract specific elements, such as the position of the hands.
In the end, it makes it possible to use even a rough sketch and turn it into a high-quality image. If you’re bad at drawing and scribble a dog quickly, simply by providing the indication “dog”, ControlNet via Scribble Diffusion takes care of transforming it into a work of art. Basically, you can draw with your mouse and the AI takes care of the rest. You can access Scribble Diffusion via this link, we can’t wait to see your achievements.
ControlNet is only three weeks old
Like recent OpenAI tools, ControlNet was launched only three weeks ago and developers are already building applications with this technology. Combined with Stable Diffusion, also open-source, these tools offer developers and machine learning enthusiasts the means to experiment with different types of models.
Other templates include Riffusion, which generates real-time AI music by converting audio frames to audio. This approach involves refining Stable Diffusion to create spectrograms, which are visualizations of audio. Another model, Amazing AI, was built on the basis of Stable Diffusion 1.5 and works natively on macOS, you don’t need an internet connection to use it. All calculations are performed locally.
Do you use Google News (News in France)? You can follow your favorite media. Follow Frandroid on Google News (and Numerama).