Modly Image-to-3D Skill Skill by ara.so — Daily 2026 Skills collection. Modly is a local, open-source desktop application (Windows/Linux) that converts photos into 3D mesh models using AI models running entirely on your GPU — no cloud, no API keys required. Architecture Overview modly/ ├── src/ # Electron + TypeScript frontend │ ├── main/ # Electron main process │ ├── renderer/ # React UI (renderer process) │ └── preload/ # IPC bridge ├── api/ # Python FastAPI backend │ ├── generator.py # Core generation logic │ └── requirements.txt ├── resources/ │ └── icons/ ├── launcher.bat # Windows quick-start ├── launcher.sh # Linux quick-start └── package.json The app runs as an Electron shell over a local Python FastAPI server. Extensions are GitHub repos with a manifest.json + generator.py that plug into the extension system. Installation Quick start (no build required)
Windows
launcher.bat
Linux
chmod +x launcher.sh ./launcher.sh Development setup
1. Clone
git clone https://github.com/lightningpixel/modly cd modly
2. Install JS dependencies
npm install
3. Set up Python backend
cd api python -m venv .venv
Activate (Windows)
.venv \ Scripts \ activate
Activate (Linux/macOS)
source .venv/bin/activate pip install -r requirements.txt cd ..
4. Run dev mode (starts Electron + Python backend)
npm run dev Production build
Build installers for current platform
npm run build
Output goes to dist/
Key npm Scripts npm run dev
Start app in development mode (hot reload)
npm run build
Package app for distribution
npm run lint
Run ESLint
npm run typecheck
TypeScript type checking
Extension System Extensions are GitHub repositories containing: manifest.json — metadata and model variants generator.py — generation logic implementing the Modly extension interface manifest.json structure { "name" : "My 3D Extension" , "id" : "my-extension-id" , "description" : "Generates 3D models using XYZ model" , "version" : "1.0.0" , "author" : "Your Name" , "repository" : "https://github.com/yourname/my-modly-extension" , "variants" : [ { "id" : "model-small" , "name" : "Small (faster)" , "description" : "Lighter variant for faster generation" , "size_gb" : 4.2 , "vram_gb" : 6 , "files" : [ { "url" : "https://huggingface.co/yourorg/yourmodel/resolve/main/weights.safetensors" , "filename" : "weights.safetensors" , "sha256" : "abc123..." } ] } ] } generator.py interface
api/extensions//generator.py
Required interface every extension must implement
import sys import json from pathlib import Path def generate ( image_path : str , output_path : str , variant_id : str , models_dir : str , ** kwargs ) -
dict : """ Required entry point for all Modly extensions. Args: image_path: Path to input image file output_path: Path where output .glb/.obj should be saved variant_id: Which model variant to use models_dir: Directory where downloaded model weights live Returns: dict with keys: success (bool) output_file (str) — path to generated mesh error (str, optional) """ try :
Load your model weights
weights
Path ( models_dir ) / variant_id / "weights.safetensors"
Run your inference
mesh
run_inference ( str ( weights ) , image_path )
Save output
mesh . export ( output_path ) return { "success" : True , "output_file" : output_path } except Exception as e : return { "success" : False , "error" : str ( e ) } Installing an extension (UI flow) Open Modly → go to Models page Click Install from GitHub Paste the HTTPS URL, e.g. https://github.com/lightningpixel/modly-hunyuan3d-mini-extension After install, click Download on the desired model variant Select the installed model and upload an image to generate Official Extensions Extension Model modly-hunyuan3d-mini-extension Hunyuan3D 2 Mini Python Backend API (FastAPI) The backend runs locally. Key endpoints used by the Electron frontend:
Typical backend route patterns (api/main.py or similar)
GET /extensions — list installed extensions
GET /extensions/{id} — get extension details + variants
POST /extensions/install — install extension from GitHub URL
POST /generate — trigger 3D generation
GET /generate/status — poll generation progress
GET /models — list downloaded model variants
POST /models/download — download a model variant
Calling the backend from Electron (IPC pattern) // src/preload/index.ts — exposing backend calls to renderer import { contextBridge , ipcRenderer } from 'electron' contextBridge . exposeInMainWorld ( 'modly' , { generate : ( imagePath : string , extensionId : string , variantId : string ) => ipcRenderer . invoke ( 'generate' , { imagePath , extensionId , variantId } ) , installExtension : ( repoUrl : string ) => ipcRenderer . invoke ( 'install-extension' , { repoUrl } ) , listExtensions : ( ) => ipcRenderer . invoke ( 'list-extensions' ) , } ) // src/main/ipc-handlers.ts — main process handling import { ipcMain } from 'electron' ipcMain . handle ( 'generate' , async ( _event , { imagePath , extensionId , variantId } ) => { const response = await fetch ( 'http://localhost:PORT/generate' , { method : 'POST' , headers : { 'Content-Type' : 'application/json' } , body : JSON . stringify ( { image_path : imagePath , extension_id : extensionId , variant_id : variantId } ) , } ) return response . json ( ) } ) // src/renderer/components/GenerateButton.tsx — UI usage declare global { interface Window { modly : { generate : ( imagePath : string , extensionId : string , variantId : string ) => Promise < { success : boolean ; output_file ? : string ; error ? : string }
installExtension : ( repoUrl : string ) => Promise < { success : boolean }
listExtensions : ( ) => Promise < Extension [ ]
} } } async function handleGenerate ( imagePath : string ) { const result = await window . modly . generate ( imagePath , 'modly-hunyuan3d-mini-extension' , 'hunyuan3d-mini-turbo' ) if ( result . success ) { console . log ( 'Mesh saved to:' , result . output_file ) } else { console . error ( 'Generation failed:' , result . error ) } } Writing a Custom Extension Minimal extension repository structure my-modly-extension/ ├── manifest.json └── generator.py Example: wrapping a HuggingFace diffusion model
generator.py
import torch from PIL import Image from pathlib import Path def generate ( image_path , output_path , variant_id , models_dir , ** kwargs ) : device = "cuda" if torch . cuda . is_available ( ) else "cpu" weights_dir = Path ( models_dir ) / variant_id try :
Load model (example pattern)
from your_model_lib import ImageTo3DPipeline pipe = ImageTo3DPipeline . from_pretrained ( str ( weights_dir ) , torch_dtype = torch . float16 ) . to ( device ) image = Image . open ( image_path ) . convert ( "RGB" ) with torch . no_grad ( ) : mesh = pipe ( image ) . mesh mesh . export ( output_path ) return { "success" : True , "output_file" : output_path } except Exception as e : return { "success" : False , "error" : str ( e ) } Configuration & Environment Modly runs fully locally — no environment variables or API keys needed. GPU/CUDA is auto-detected by PyTorch in extensions. Relevant configuration lives in: package.json # Electron app metadata, build targets api/requirements.txt # Python dependencies for backend If you need to configure the backend port or extension directory, check the Electron main process config (typically src/main/index.ts ) for constants like API_PORT or EXTENSIONS_DIR . Common Patterns Check if CUDA is available in an extension import torch def get_device ( ) : if torch . cuda . is_available ( ) : print ( f"Using GPU: { torch . cuda . get_device_name ( 0 ) } " ) return "cuda" print ( "No GPU found, falling back to CPU (slow)" ) return "cpu" Progress reporting from generator.py import sys import json def report_progress ( percent : int , message : str ) : """Write progress to stdout so Modly can display it.""" print ( json . dumps ( { "progress" : percent , "message" : message } ) , flush = True ) def generate ( image_path , output_path , variant_id , models_dir , ** kwargs ) : report_progress ( 0 , "Loading model..." )
... load model ...
report_progress ( 30 , "Processing image..." )
... inference ...
report_progress ( 90 , "Exporting mesh..." )
... export ...
report_progress ( 100 , "Done" ) return { "success" : True , "output_file" : output_path } Adding a new page in the renderer (React) // src/renderer/pages/MyPage.tsx import React , { useEffect , useState } from 'react' interface Extension { id : string name : string description : string } export default function MyPage ( ) { const [ extensions , setExtensions ] = useState < Extension [ ]
( [ ] ) useEffect ( ( ) => { window . modly . listExtensions ( ) . then ( setExtensions ) } , [ ] ) return ( < div
< h1
Installed Extensions < / h1
{ extensions . map ( ext => ( < div key = { ext . id }
< h2
{ ext . name } < / h2
< p
{ ext . description } < / p
< / div
) ) } < / div
) } Troubleshooting Problem Fix npm run dev — Python backend not starting Ensure venv is set up: cd api && python -m venv .venv && pip install -r requirements.txt CUDA out of memory Use a smaller model variant or close other GPU processes Extension install fails Verify the GitHub URL is HTTPS and the repo contains manifest.json at root Generation hangs Check that your GPU drivers and CUDA toolkit match the PyTorch version in requirements.txt App won't launch on Linux Make launcher.sh executable: chmod +x launcher.sh Model download stalls Check disk space; large models (4–10 GB) need adequate free space torch not found in extension Ensure PyTorch is in api/requirements.txt , not just the extension's own deps Verifying GPU is detected cd api source .venv/bin/activate
or .venv\Scripts\activate on Windows
- python
- -c
- "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'no GPU')"
- Resources
- Homepage
- :
- https://modly3d.app
- Releases
- :
- https://github.com/lightningpixel/modly/releases/latest
- Official extension
- :
- https://github.com/lightningpixel/modly-hunyuan3d-mini-extension
- Discord
- :
- https://discord.gg/FjzjRgweVk
- License
- MIT (attribution required — credit Modly + Lightning Pixel in forks)