唠唠闲话

整理一些优秀的项目。

Ollama: 本地部署的语言模型服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: '3.8'
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - /sshfs/ollama:/root/.ollama
    environment:
      - CUDA_VISIBLE_DEVICES=0,1
      # - OLLAMA_HOST="http://localhost:11434" # 监听端口和地址
      - OLLAMA_KEEP_ALIVE="5m"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

并发硬伤：https://github.com/ollama/ollama/issues/358
持续更新中：https://github.com/ollama/ollama/pull/3418

Open-WebUI: https://github.com/open-webui/open-webui

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
version: '3.8'
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: always
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL: ${OLLAMA_BASE_URL}
      - RAG_EMBEDDING_MODEL: "all-MiniLM-L6-v2"
      - RAG_EMBEDDING_MODEL_DIR: "/app/backend/data/cache/embedding/models/"
      - HF_ENDPOINT: https://hf-mirror.com
    volumes:
      - ./open-webui:/app/backend/data

LiteLLM: https://github.com/BerriAI/litellm