GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
- Updated
May 27, 2025 - C++
llm-inference 开源项目推荐（4,872 个）
相关标签
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
python data-science machine-learning reinforcement-learning deep-learning deployment tensorflow optimization parallel pytorch distributed hyperparameter-optimization ray hyperparameter-search serving rllib large-language-models llm llm-serving llm-inference
- Updated
Nov 1, 2026 - Python
Find secrets with Gitleaks 🔑
git go cli golang open-source security secret ci-cd cicd hacktoberfest dlp security-tools devsecops data-loss-prevention gitleaks ai-powered nhi llm llm-training llm-inference
- Updated
Oct 28, 2026 - Go
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
- Updated
Oct 14, 2026 - HTML
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
- Updated
Nov 1, 2026 - Python
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
llama mistral fine-tuning mlops bentoml vicuna llm model-inference llmops llm-serving llm-inference open-source-llm llama2 openllm llm-ops llama3-1 llama3-2 llama3-2-vision
- Updated
Oct 26, 2026 - Python
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
nlp natural-language-processing ai computer-vision deep-learning transformers inference speech-recognition yolo recommendation-system performance-boost good-first-issue openvino diffusion-models stable-diffusion generative-ai llm-inference optimize-ai deploy-ai
- Updated
Nov 1, 2026 - C++
Official inference library for Mistral models
- Updated
Jun 16, 2026 - Jupyter Notebook
High-speed Large Language Model Serving for Local Deployment
- Updated
May 11, 2026 - C++
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
machine-learning deep-learning c99 simd transformer moe avx2 quantization memory-efficient from-scratch systems-programming inference-engine mixture-of-experts zero-dependencies linear-attention cpu-inference llm llm-inference mxfp4 kimi-k3
- Updated
Oct 2, 2026 - C
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform
- Updated
Oct 29, 2026 - Python
A Datacenter Scale Distributed Inference Serving Framework
kubernetes rust routing-engine omni diffusion vllm llm-inference tensorrt-llm sglang disaggregated-serving
- Updated
Nov 1, 2026 - Rust
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
llama cuda-kernels deepspeed llm fastertransformer llm-inference turbomind internlm llama2 codellama llama3
- Updated
Oct 30, 2026 - Python
Open-source implementation of AlphaEvolve
genetic-algorithm discovery optimize evolutionary-algorithms deepmind-lab deepmind iterative-methods genetic-algorithms evolutionary-computation alphacode distributed-evolutionary-algorithms iterative-refinement llm-inference llm-engineering llm-ensemble coding-agent alpha-evolve alphaevolve openevolve
- Updated
Nov 1, 2026 - Python
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
distributed-systems performance-engineering book gpu accelerator mixture-of-experts kv-cache ai-infra llm llm-serving llm-training llm-inference deepseek ai-infrastructure datacenter-network
- Updated
Oct 1, 2026 - Python
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
proxy routing gateway prompt proxy-server openai envoy envoyproxy llms generative-ai llmops llm-inference llm-proxy ai-gateway llm-gateway llm-routing ai-gateway-support
- Updated
Oct 24, 2026 - Rust
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
macos swift metal gpgpu gemma on-device-ai apple-silicon llm llm-inference local-ai on-device-llm gemma4 gemma4-26b-a4b
- Updated
Oct 26, 2026 - Swift
FlashInfer: Kernel Library for LLM Serving
- Updated
Nov 1, 2026 - Cuda
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
kubernetes machine-learning tensorflow cncf pytorch artificial-intelligence xgboost k8s service-mesh hacktoberfest istio model-serving kubeflow mlops knative model-interpretability kserve vllm genai llm-inference
- Updated
Oct 31, 2026 - Go
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
android ios arm mobile framework ai speech edge transformer smartphone whisper rag edge-ai on-device-ai mobile-inference quantiz llm llms llamacpp llm-inference
- Updated
Sep 26, 2026 - C++