V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
• 请不要在回答技术问题时复制粘贴 AI 生成的内容
RoxyBrowser - 轻盈、安全的指纹浏览器
Roxy 凭借领先的指纹反追踪技术,助力隐私保护、跨境电商、数据采集等场景,为您的卓越事业保驾护航。
Promoted by RoxyBrowser
wukaige
V2EX  ›  程序员

求助, Milvus 数据库导入数据会导致数据库崩溃,是我的配置问题吗?

  •  
  •   wukaige · 7 天前 · 691 次点击

    数据量:6000w

    Milvus docker compose:

    services:
      etcd:
        container_name: milvus-etcd
        image: quay.io/coreos/etcd:v3.5.18
        environment:
          - ETCD_AUTO_COMPACTION_MODE=revision
          - ETCD_AUTO_COMPACTION_RETENTION=1000
          - ETCD_QUOTA_BACKEND_BYTES=8589934592
          - ETCD_SNAPSHOT_COUNT=50000
          - ETCD_MAX_REQUEST_BYTES=33554432
        volumes:
          - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
        command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
        healthcheck:
          test: ["CMD", "etcdctl", "endpoint", "health"]
          interval: 30s
          timeout: 20s
          retries: 3
        ulimits:
          nofile:
            soft: 655360
            hard: 655360
        mem_limit: 16g
        cpus: 4.0
        logging:
          driver: "json-file"
          options:
            max-size: "100m"
            max-file: "3"
    
      minio:
        container_name: milvus-minio
        image: minio/minio:RELEASE.2024-05-28T17-19-04Z
        environment:
          MINIO_ACCESS_KEY: xxxxx
          MINIO_SECRET_KEY: xxxxx
        ports:
          - "9001:9001"
          - "9000:9000"
        volumes:
          - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
        command: minio server /minio_data --console-address ":9001"
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
        ulimits:
          nofile:
            soft: 655360
            hard: 655360
        mem_limit: 16g
        cpus: 4.0
        logging:
          driver: "json-file"
          options:
            max-size: "100m"
            max-file: "3"
    
      standalone:
        container_name: milvus
        image: milvusdb/milvus:v2.6.0-rc1
        command: ["milvus", "run", "standalone"]
        security_opt:
        - seccomp:unconfined
        environment:
          ETCD_ENDPOINTS: etcd:2379
          MINIO_ADDRESS: minio:9000
          MQ_TYPE: woodpecker
        volumes:
          - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/_milvus:/var/lib/milvus
          - ./milvus.yaml:/milvus/configs/milvus.yaml
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
          interval: 30s
          start_period: 90s
          timeout: 20s
          retries: 3
        ports:
          - "xxxx:19530"
          - "xxxx:9091"
        depends_on:
          - "etcd"
          - "minio"
        ulimits:
          nofile:
            soft: 655360
            hard: 655360
        mem_limit: 1024g
        cpus: 32.0
        logging:
          driver: "json-file"
          options:
            max-size: "100m"
            max-file: "3"
    
    networks:
      default:
        name: milvus
    

    机器配置:4T 内存,256 CPU

    milvus.yaml: https://raw.githubusercontent.com/milvus-io/milvus/v2.6.0-rc1/configs/milvus.yaml

    部分日志:

    milvus        | [2025/07/18 15:41:28.119 +00:00] [WARN] [timetick/timetick_sync_operator.go:85] ["send time tick sync message failed"] [module=streamingnode] [component=timetick-sync] [pchannel=by-dev-rootcoord-dml_8:rw@3] [error="append time tick msg to wal failed, timestamp: 459499972358307846, previous message counter: 8: code: STREAMING_CODE_CHANNEL_FENCED, cause: by-dev-rootcoord-dml_8:rw@3 fenced"]
    milvus        | [2025/07/18 15:41:28.119 +00:00] [WARN] [timetick/timetick_sync_operator.go:85] ["send time tick sync message failed"] [module=streamingnode] [component=timetick-sync] [pchannel=by-dev-rootcoord-dml_14:rw@3] [error="append time tick msg to wal failed, timestamp: 459499972358307848, previous message counter: 8: code: STREAMING_CODE_CHANNEL_FENCED, cause: by-dev-rootcoord-dml_14:rw@3 fenced"]
    milvus        | [2025/07/18 15:41:28.119 +00:00] [WARN] [timetick/timetick_sync_operator.go:85] ["send time tick sync message failed"] [module=streamingnode] [component=timetick-sync] [pchannel=by-dev-rootcoord-dml_7:rw@3] [error="append time tick msg to wal failed, timestamp: 459499972358307847, previous message counter: 8: code: STREAMING_CODE_CHANNEL_FENCED, cause: by-dev-rootcoord-dml_7:rw@3 fenced"]
    milvus        | [2025/07/18 15:41:28.120 +00:00] [WARN] [sessionutil/session_util.go:593] ["fail to retry keepAliveOnce"] [serverName=querynode] [LeaseID=7587888197442225626] [error="etcdserver: requested lease not found"]
    milvus        | [2025/07/18 15:41:28.121 +00:00] [ERROR] [querynodev2/server.go:188] ["Query Node disconnected from etcd, process will exit"] ["Server Id"=2] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Register.func1\n\t/workspace/source/internal/querynodev2/server.go:188"]
    milvus        | [2025/07/18 15:41:28.121 +00:00] [WARN] [sessionutil/session_util.go:593] ["fail to retry keepAliveOnce"] [serverName=mixcoord] [LeaseID=7587888197442225598] [error="etcdserver: requested lease not found"]
    milvus        | [2025/07/18 15:41:28.122 +00:00] [ERROR] [coordinator/mix_coord.go:107] ["MixCoord disconnected from etcd, process will exit"] [serverID=2] [stack="github.com/milvus-io/milvus/internal/coordinator.(*mixCoordImpl).Register.(*mixCoordImpl).Register.func1.func3\n\t/workspace/source/internal/coordinator/mix_coord.go:107"]
    milvus        | [2025/07/18 15:41:28.122 +00:00] [WARN] [sessionutil/session_util.go:593] ["fail to retry keepAliveOnce"] [serverName=proxy] [LeaseID=7587888197442225923] [error="etcdserver: requested lease not found"]
    milvus        | [2025/07/18 15:41:28.122 +00:00] [ERROR] [proxy/proxy.go:181] ["Proxy disconnected from etcd, process will exit"] ["Server Id"=2] [stack="github.com/milvus-io/milvus/internal/proxy.(*Proxy).Register.func1\n\t/workspace/source/internal/proxy/proxy.go:181"]
    milvus        | [2025/07/18 15:41:28.122 +00:00] [WARN] [handler/handler_client_impl.go:178] ["create handler failed"] [pchannel=by-dev-rootcoord-dml_10] [handler=producer] [assignment=by-dev-rootcoord-dml_10:rw@3>2@172.23.0.4:22222] [error="/milvus.proto.streaming.StreamingNodeHandlerService/Produce; streaming error: code = STREAMING_CODE_CHANNEL_NOT_EXIST, cause = by-dev-rootcoord-dml_10 not exist; rpc error: code = FailedPrecondition, desc = "]
    milvus        | [2025/07/18 15:41:28.123 +00:00] [INFO] [handler/handler_client_impl.go:183] ["report assignment error"] [pchannel=by-dev-rootcoord-dml_10] [handler=producer] [assignmentError="/milvus.proto.streaming.StreamingNodeHandlerService/Produce; streaming error: code = STREAMING_CODE_CHANNEL_NOT_EXIST, cause = by-dev-rootcoord-dml_10 not exist; rpc error: code = FailedPrecondition, desc = "] []
    milvus        | [2025/07/18 15:41:28.120 +00:00] [ERROR] [streamingnode/service.go:389] ["StreamingNode disconnected from etcd, process will exit"] ["Server Id"=2] [stack="github.com/milvus-io/milvus/internal/distributed/streamingnode.(*Server).registerSessionToETCD.func1\n\t/workspace/source/internal/distributed/streamingnode/service.go:389"]
    milvus        | [2025/07/18 15:41:28.120 +00:00] [ERROR] [datanode/data_node.go:200] ["Data Node disconnected from etcd, process will exit"] ["Server Id"=2] [stack="github.com/milvus-io/milvus/internal/datanode.(*DataNode).Register.func1\n\t/workspace/source/internal/datanode/data_node.go:200"]
    
    defunct9
        1
    defunct9  
       7 天前
    开 ssh ,让我上去看看
    wukaige
        2
    wukaige  
    OP
       7 天前
    @defunct9

    不行 :<
    juzisang
        3
    juzisang  
       7 天前
    开 ssh ,让楼上上去看看
    smplesugar
        4
    smplesugar  
       7 天前
    开 ssh ,让楼上上上去看看
    maocat
        5
    maocat  
       7 天前
    去 github 上贴 issue 啊,不过你这有点勇,上来就整 2.6 的 rc 版本,2.5.X 现在还在往后走着呢
    wukaige
        6
    wukaige  
    OP
       7 天前
    @maocat

    2.5.x 也有这个问题
    guanzhangzhang
        7
    guanzhangzhang  
       7 天前
    你这频繁报错[error="etcdserver: requested lease not found"],先看下 etcd 日志
    xiaoheijw
        8
    xiaoheijw  
       7 天前   ❤️ 1
    https://milvus.io/zh/community
    可以去 milvus 微信社区群问问,专家在线答疑
    关于   ·   帮助文档   ·   自助推广系统   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1043 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 26ms · UTC 22:46 · PVG 06:46 · LAX 15:46 · JFK 18:46
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.