mlx-local-inference

Category: Data & AI | Uploader: bendusy | Downloads: 0 | Version: v1.0（Latest）

Use when calling local AI on this Mac — text generation, embeddings, speech-to-text, OCR, or image understanding. LLM/VLM via oMLX gateway at

Changelog: Source: GitHub https://github.com/bendusy/mlx-local-inference

Directory Structure

Current level: Root

📁 references/
- 📄 asr-qwen3.md 2.0 KB
- 📄 asr-whisper.md 1.6 KB
- 📄 embedding-qwen3.md 2.1 KB
- 📄 llm-gemma3-12b.md 1.8 KB
- 📄 llm-models-reference.md 1.5 KB
- 📄 llm-qwen3-14b.md 2.9 KB
- 📄 ocr.md 2.7 KB
- 📄 omlx.md 1.7 KB
- 📄 transcribe-daemon.md 1.9 KB
- 📄 tts-qwen3.md 1.1 KB
📁 server/
- 📄 admin_api_patch.py 4.9 KB
- 📄 config.yaml 1.4 KB
- 📄 idle-unload-watchdog.py 8.6 KB
- 📄 lazy_handler_proxy.py 2.8 KB
- 📄 start_with_admin.py 3.8 KB
- 📄 transcribe-daemon.py 16.1 KB
- 📄 watchdog.yaml 696 B
📄 .gitignore 41 B
📄 LICENSE 1.0 KB
📄 README.md 6.1 KB
📄 README_CN.md 6.8 KB
📄 SKILL.md 6.3 KB

SKILL.md

---
name: mlx-local-inference
description: >
  Use when calling local AI on this Mac — text generation, embeddings,
  speech-to-text, OCR, or image understanding. LLM/VLM via oMLX gateway at
  localhost:8000/v1. Embedding/ASR/OCR via Python libraries (mlx-lm, mlx-vlm).
  Works offline. Use instead of cloud APIs for privacy or low latency.
metadata: { "openclaw": { "os": ["darwin"], "requires": { "anyBins": ["python3"] } } }
---

# MLX Local Inference Stack

Local AI inference on Apple Silicon. **oMLX** handles LLM/VLM with continuous batching.
Python libraries handle Embedding/ASR/OCR directly.

## Architecture

```
┌─────────────────────────────────────┐
│  oMLX (localhost:8000/v1)           │
│  - LLM (Qwen3-14B, etc.)            │
│  - VLM (vision-language models)     │
│  - Continuous batching + SSD cache  │
└─────────────────────────────────────┘

┌─────────────────────────────────────┐
│  Python Libraries (direct call)     │
│  - mlx-lm: Embedding                │
│  - mlx-vlm: OCR (PaddleOCR-VL)      │
│  - mlx-whisper: ASR (Qwen3-ASR)     │
└─────────────────────────────────────┘
```

## Models

| Capability | Implementation | Model | Size |
|-----------|---------------|-------|------|
| 💬 LLM | oMLX API | `Qwen3-14B-4bit` | ~8 GB |
| 👁️ VLM | oMLX API | Any mlx-vlm model | varies |
| 📐 Embed | mlx-lm (Python) | `Qwen3-Embedding-0.6B-4bit-DWQ` | ~1 GB |
| 🎤 ASR | mlx-whisper (Python) | `Qwen3-ASR-1.7B-8bit` | ~1.5 GB |
| 👁️ OCR | mlx-vlm (Python) | `PaddleOCR-VL-1.5-6bit` | ~3.3 GB |

## Usage

### LLM / Vision-Language (via oMLX API)

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

# Text generation
resp = client.chat.completions.create(
    model="Qwen3-14B-4bit",
    messages=[{"role": "user", "content": "Hello"}]
)
print(resp.choices[0].message.content)

# Vision (image + text)
import base64
with open("image.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="Qwen3-14B-4bit",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
            {"type": "text", "text": "What is in this image?"}
        ]
    }]
)
```

---

### Embeddings (via mlx-lm Python library)

```python
from mlx_lm import load, generate

# Load from ~/models/ (oMLX-compatible path)
model, tokenizer = load("~/models/Qwen3-Embedding-0.6B-4bit-DWQ")

# Get embeddings
text = "text to embed"
inputs = tokenizer(text, return_tensors="np")
embeddings = model(**inputs).last_hidden_state.mean(axis=1)
print(embeddings.shape)  # (1, 1024)
```

---

### ASR — Speech-to-Text (via mlx-whisper Python library)

```python
import mlx_whisper

# Transcribe audio (load from ~/models/)
result = mlx_whisper.transcribe(
    "audio.wav",
    path_or_hf_repo="~/models/Qwen3-ASR-1.7B-8bit",
    language="zh"  # or "en", omit for auto-detect
)
print(result["text"])
```

Supported formats: `wav`, `mp3`, `m4a`, `flac`, `ogg`, `webm`

---

### OCR (via mlx-vlm Python library)

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_image

# Load from ~/models/ (oMLX-compatible path)
model, processor = load("~/models/PaddleOCR-VL-1.5-6bit")
image = load_image("document.jpg")

prompt = apply_chat_template(
    processor, 
    config=model.config, 
    prompt="OCR:",
    num_images=1
)

output = generate(
    model, 
    processor, 
    image, 
    prompt, 
    max_tokens=512,
    temp=0.0
)
print(output)
```

> **Note:** For OCR, prompt must be `"OCR:"` and temperature `0.0`.

---

## Service Management (oMLX only)

```bash
# Check running models
curl http://localhost:8000/v1/models

# Restart oMLX
launchctl kickstart -k gui/$(id -u)/com.omlx-server

# Logs
tail -f /tmp/omlx-server.log

# Manual start (debug)
omlx serve --model-dir ~/models --port 8000
```

## Installation

```bash
# 1. Install oMLX for LLM/VLM (via Homebrew - recommended)
brew tap omlx-ai/tap
brew install omlx

# 2. Install Python libraries for Embedding/ASR/OCR
pip install mlx-lm mlx-vlm mlx-whisper huggingface_hub
```

**Note:** If you don't have Homebrew, install it first: `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"`

## Setup

```bash
git clone https://github.com/bendusy/mlx-local-inference
cd mlx-local-inference

# Install dependencies
pip install mlx-lm mlx-vlm mlx-whisper huggingface_hub

# Download models to ~/models/ (oMLX-compatible structure)
python3 -c "
from huggingface_hub import snapshot_download
import os

models = [
    'mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ',
    'mlx-community/Qwen3-ASR-1.7B-8bit',
    'mlx-community/PaddleOCR-VL-1.5-6bit',
    'mlx-community/Qwen3-14B-4bit'
]

for repo_id in models:
    model_name = repo_id.split('/')[-1]
    local_dir = os.path.expanduser(f'~/models/{model_name}')
    print(f'Downloading {model_name}...')
    snapshot_download(
        repo_id=repo_id,
        local_dir=local_dir,
        local_dir_use_symlinks=False
    )
"

# Install and start oMLX (for LLM/VLM)
brew tap omlx-ai/tap
brew install omlx
omlx serve --model-dir ~/models --port 8000
```

## Model Storage Strategy

**All models stored in `~/models/` using oMLX-compatible structure:**

```
~/models/
├── Qwen3-Embedding-0.6B-4bit-DWQ/
│   ├── config.json
│   └── *.safetensors
├── Qwen3-ASR-1.7B-8bit/
├── PaddleOCR-VL-1.5-6bit/
└── Qwen3-14B-4bit/
```

**Why this structure:**
- oMLX requires models in `~/models/<ModelName>/` format
- Python libraries can load from local paths (`~/models/...`)
- **Future-proof:** When oMLX adds Embedding/ASR support, we can switch instantly without moving models

**Migration path:**
- Currently: Python libs load from `~/models/` directly
- Future: When oMLX supports these models, switch to oMLX API endpoints
- No model re-download needed

## Requirements

- Apple Silicon Mac (M1/M2/M3/M4)
- macOS 13+, Python 3.10+
- 16 GB RAM minimum (32 GB for 35B models)

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

mlx-local-inference

Directory Structure

SKILL.md

Report

Notice