这是一个创建于 1385 天前的主题，其中的信息可能已经有所发展或是发生改变。
是这样的，今天发现 k8s 有 2 台 master 节点 CPU 使用率有尖峰（ k8s 1.11 Linux 3.10 1.13.1 ），由于没有当时进程状态快照，只能用 top 每秒抓一次进程扔日志里。果然过了一段时间，发现了几个叫 docker-containerd-current 的进程 cpu 飙升导致 CPU 使用率高达 90%，但是我们通过 ps pgrep 等等手段都查不到这个进程了，所以考虑是不是当时突然启动了这些进程然后又死了，后来想从 /proc 里查有没有啥信息，奇怪的事发生了：ll -arth /proc|grep $PID 通过 pid 号对应文件夹查不到这个进程，但是 cd $PID 居然可以进到进程的文件夹里，然后我不死心，退回到 /proc 里又肉眼看了一遍文件夹，还是没找到。进到 /proc/$PID 内查看了 exe 对应的二进制文件，然后通过 ps -ef|grep $EXE_FILE 发现对应的进程启动时间还是去年。
这里的问题就是 /proc 下面的以 pid 命名的文件夹不应该是正在运行着的进程的么？进程确实死了，通过文件名也查不到了，但是为什么可以进到一个根本不存在的文件夹呢？
重点是图里面圈中的部分。
这里的问题就是 /proc 下面的以 pid 命名的文件夹不应该是正在运行着的进程的么？进程确实死了，通过文件名也查不到了，但是为什么可以进到一个根本不存在的文件夹呢？
重点是图里面圈中的部分。
 |
1
mikeguan 2021-02-07 18:01:37 +08:00 via Android
ls 显示目录的参数是-d
|
 |
3
lcdtyph 2021-02-07 18:07:49 +08:00 via iPhone
ll 是什么的 alias
|
 |
6
zhoudaiyu OP @mikeguan 但是问题是查不到文件夹为啥可以进去，而且进程都死了，不应该在 proc 下了啊
|
 |
9
Jirajine 2021-02-07 18:26:38 +08:00 via Android
可能是进程被隐藏了。
|
|
12
Jirajine 2021-02-07 18:32:23 +08:00 via Android  1
|
|
13
lcdtyph 2021-02-07 18:38:12 +08:00 via iPhone 3
@zhoudaiyu
如果没有什么特殊的 mount options，那说明你这个不是 pid，而是某个 tid，你需要找到它的主进程再看。 /proc/<tid> 是不会出现在 getdents 调用中，所以不会被 ls 看到 |
|
15
zhoudaiyu OP @lcdtyph 老哥 我看有 mountinfo mounts mountstats 截图里是 mountstats 的输出，tid 是线程 id 吗？
device rootfs mounted on / with fstype rootfsdevice sysfs mounted on /sys with fstype sysfs device proc mounted on /proc with fstype procdevice devtmpfs mounted on /dev with fstype devtmpfs device securityfs mounted on /sys/kernel/security with fstype securityfsdevice tmpfs mounted on /dev/shm with fstype tmpfsdevice devpts mounted on /dev/pts with fstype devptsdevice tmpfs mounted on /run with fstype tmpfs device tmpfs mounted on /sys/fs/cgroup with fstype tmpfsdevice cgroup mounted on /sys/fs/cgroup/systemd with fstype cgroupdevice pstore mounted on /sys/fs/pstore with fstype pstoredevice cgroup mounted on /sys/fs/cgroup/pids with fstype cgroup device cgroup mounted on /sys/fs/cgroup/blkio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/perf_event with fstype cgroup device cgroup mounted on /sys/fs/cgroup/freezer with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/net_cls,net_prio with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/devices with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/cpu,cpuacct with fstype cgroup device cgroup mounted on /sys/fs/cgroup/hugetlb with fstype cgroup device cgroup mounted on /sys/fs/cgroup/cpuset with fstype cgroupdevice cgroup mounted on /sys/fs/cgroup/memory with fstype cgroup device configfs mounted on /sys/kernel/config with fstype configfsdevice /dev/mapper/vg_root-lv_root mounted on / with fstype xfsdevice selinuxfs mounted on /sys/fs/selinux with fstype selinuxfsdevice systemd-1 mounted on /proc/sys/fs/binfmt_misc with fstype autofs device hugetlbfs mounted on /dev/hugepages with fstype hugetlbfsdevice debugfs mounted on /sys/kernel/debug with fstype debugfs device mqueue mounted on /dev/mqueue with fstype mqueue device /dev/sda1 mounted on /boot with fstype xfsdevice /dev/mapper/vg_root-lv_var mounted on /var with fstype xfsdevice /dev/mapper/vg_root-lv_opt mounted on /opt with fstype xfsdevice /dev/mapper/vg_app-lv_DATA mounted on /DATA with fstype xfs device /dev/mapper/vg_root-lv_tmp mounted on /tmp with fstype xfs device /dev/mapper/vg_root-lv_home mounted on /home with fstype xfs device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/containers with fstype xfs device /dev/mapper/vg_root-lv_var mounted on /var/lib/docker/overlay2 with fstype xfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/206f7c4c-52cd-11ea-9f39-0050569a04bf/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/e3695fa7-2dce-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sdn-token-d66w8 with fstype tmpfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/5abfd92e-2dc8-11ea-97bb-005056a99505/volumes/kubernetes.io~secret/sync-token-tr7f8 with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/c47caa96460e233c36b6790b1ca4ea450dee1d297f9e40bf199c6ab83bb1e690/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/edb026a3c373d3ddbd064b9d8e997ccb441d56eb883f19237374bdd839cd876a/merged with fstype overlay device proc mounted on /run/docker/netns/default with fstype proc device shm mounted on /var/lib/docker/containers/6d3e50e348e7bc4608b568fd43ad63670c57f0c53de71f446ba17e8ec5e47f4e/shm with fstype tmpfs device shm mounted on /var/lib/docker/containers/f2c824ce425cfe15dda053af70d4fd02d64afdae9367ae78bcd6b5758338182c/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/60fffaf1c2c9cbe0d12504a67a0b603c9bdbf8b0f6322dc30b2befa9cbb87438/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/0045f1ad6a6556324b91b80892dcc4951ad32ec8dfe153b7c866b942fae74843/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/ec0204696ace63e3c1eb69bd81e029c13d7c1320b16b561ddceb3c56f4ae9172/merged with fstype overlay device shm mounted on /var/lib/docker/containers/a774eddee040f246b9dea20a33781a1b8cb80a3bc668690d66baec33a0f56a1d/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/cfcfc406430dc094d2b2a7084f38ea44b1eb2200282ce5fa432d46c9fee9d1f6/merged with fstype overlay device shm mounted on /var/lib/docker/containers/6d7a77c6a7dade24e619f63aa813962ae003e52f15dc146141af3c362564fd27/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/52230e78851c2d66d058e1fda52b3cb8b3493a86804354d234fe64895b3aed56/merged with fstype overlay device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/a684975d-9ef2-11ea-90f1-005056a9a751/volumes/kubernetes.io~secret/node-exporter-token-mttqc with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/c07d51282d5db3eeaa6966bdb277d34fdab0707900b34822b5bb85c6f3dddc84/merged with fstype overlay device shm mounted on /var/lib/docker/containers/14c0c7e261e57a3eac08de62225b5b33fe8a11d231b13e34ddbdec8231e98d24/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/7c510eedb0ebb50ace710478ceca3037011a99a24ded9b2862d2700064782b22/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/be29c4a1f9bbd28181851f27df38bf277a1984595490599db5d4297de8d0847f/merged with fstype overlay device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/oauth-config with fstype tmpfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/42a3edd4-02be-11eb-854a-005056a958e5/volumes/kubernetes.io~secret/default-token-mlq5l with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/2a50c3d9829ae039bbe3afb1108e2b131aca83febf40070b0423761aee576354/merged with fstype overlay device shm mounted on /var/lib/docker/containers/4b04ef977813ab3b24704d433d09d56fd069ee9103a32c5d041a3a072eab4663/shm with fstype tmpfs device proc mounted on /run/docker/netns/4dd48aec066f with fstype proc device overlay mounted on /var/lib/docker/overlay2/e1ee93593bab0e24a711c4ce871534fefe651e8be5ee11cf64f544e0b3e2e86f/merged with fstype overlay device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc device overlay mounted on /var/lib/docker/overlay2/e04f7635399c36249c8e9b90d5619ba593b7633275d1b2c2363d3c93d0e2f55b/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/0be6198515753a8a97b51d078ac64e1153325342221e8d77a956eecabd448d12/merged with fstype overlay device shm mounted on /var/lib/docker/containers/f54abb6462617d5910b8d41e5fa4c131be18aa59d4d227040e3f3e2ecd69d70a/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/a177d06ddec1fb54a0b04725fabe944293ca72eb4aa0bdbf4e918f828d089466/merged with fstype overlay device overlay mounted on /var/lib/docker/overlay2/0af1e5fe64c765d3e0a4d574b725829a6b6e3170b929ede21cdd5b5cb97b0fc4/merged with fstype overlay device shm mounted on /var/lib/docker/containers/284f17f11532b9750a1f15adb3c2361796c42c4e0afc220c49c0cd394c477299/shm with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/ce6e3d2eb2879de645014da2be3650c7f172634131d08718304ef106a3b4ea66/merged with fstype overlay device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/serving-cert with fstype tmpfs device tmpfs mounted on /var/lib/origin/openshift.local.volumes/pods/37a179b0-690c-11eb-b1fb-005056a9a751/volumes/kubernetes.io~secret/webconsole-token-spxpx with fstype tmpfs device overlay mounted on /var/lib/docker/overlay2/188242c0ee6187645f21d2f0195615af2d450337a461c4c197367bfaf794d5ed/merged with fstype overlay device shm mounted on /var/lib/docker/containers/5b5fb40679472f61b4b5f37caf20666291fdb38f04648710dbd23a3e952c0845/shm with fstype tmpfs device proc mounted on /run/docker/netns/feff0fd124b3 with fstype proc device overlay mounted on /var/lib/docker/overlay2/d6d7732f83a71c8c3f33a257bab98a140b5099e0ba2b1bb789c057520cf75d95/merged with fstype overlay device tmpfs mounted on /run/user/0 with fstype tmpfs |
|
17
lcdtyph 2021-02-07 19:13:09 +08:00 1
|
|
18
lcdtyph 2021-02-07 19:17:37 +08:00 1
|
|
19
zhoudaiyu OP @lcdtyph mount|grep 14546 返回空，ps -efL | grep 14546 这个好使，多谢老哥，但是我还是不太明白为啥 /proc 看不到这个 14546 这个进程号的文件夹，但是可以 cd 到 14546 中来，老哥能解个惑不？
|
|
20
lcdtyph 2021-02-07 19:47:04 +08:00 via iPhone 3
@zhoudaiyu
mount 那条命令是看 procfs 的 options 的，而且不是 grep 14546，就是 grep proc 。如果设置了 hidepid=2 那么 /proc 下就看不到其他用户的 pid 了 你这里的 14546 不是进程号而是个线程号(tid)，/proc/tid 下的内容和 /proc/pid/task/tid 下的内容是一模一样的，所以实现 procfs 的时候让这些入口不在 getdents 系统调用的结果中出现。而 ls 正是使用 getdents 实现的，所以 ls 自然看不到这些入口。 cd 依赖的是 chdir 系统调用，所以不受影响 |
 |
21
howellz 2021-02-07 19:54:11 +08:00
没细看，不过 docker 不就是利用 namespace 来隔离进程和文件系统（包括 proc ）的么？
|
 |
24
liuxu 2021-02-10 15:08:47 +08:00
@howellz 学到了，多谢大佬，学习了
$ man procfs /proc/[tid] subdirectories Each one of these subdirectories contains files and subdirectories exposing information about the thread with the corre‐ sponding thread ID. The contents of these directories are the same as the corresponding /proc/[pid]/task/[tid] directo‐ ries. The /proc/[tid] subdirectories are not visible when iterating through /proc with getdents(2) (and thus are not visible when one uses ls(1) to view the contents of /proc). 然后用 sysdig 抓 ls /proc 第一个命令行：$ sudo sysdig proc.name = ls 第二个命令行：$ ls /proc 然后第一个命令行得到： ... 333156 14:54:19.153132412 4 ls (671251) > stat 333159 14:54:19.153141644 4 ls (671251) < stat res=0 path=/proc 333160 14:54:19.153149403 4 ls (671251) > openat 333161 14:54:19.153159898 4 ls (671251) > fstat fd=-82524538700541 333162 14:54:19.153161715 4 ls (671251) < fstat res=0 333164 14:54:19.153169912 4 ls (671251) > getdents64 fd=137455746819 333175 14:54:19.154214139 4 ls (671251) < getdents64 res=11840 333176 14:54:19.154223263 4 ls (671251) > lstat 333177 14:54:19.154232665 4 ls (671251) < lstat res=0 path=/proc/fb 333178 14:54:19.154238761 4 ls (671251) > lstat 333179 14:54:19.154242851 4 ls (671251) < lstat res=0 path=/proc/fs 333180 14:54:19.154247207 4 ls (671251) > lstat 333181 14:54:19.154250957 4 ls (671251) < lstat res=0 path=/proc/bus 333182 14:54:19.154254649 4 ls (671251) > lstat 333183 14:54:19.154257514 4 ls (671251) < lstat res=0 path=/proc/dma 333184 14:54:19.154260911 4 ls (671251) > lstat 333185 14:54:19.154263837 4 ls (671251) < lstat res=0 path=/proc/i8k ... |
Select Language