sd 0:0:6:0: tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK scsi_io_completion: 6 callbacks suppressed sd 0:0:6:0: device_unblock and setting to running, handle(0x0010) Kern.log from when a disk actually disconnects sd 0:0:6:0: device_block, handle(0x0010) It seems like there should be something I can do since if I kill it early enough it can always get caught up before any i/o operations time out and disconnects a disk. This is a big problem because if I don't kill the load script when this starts happening, eventually i/o operations time out causing the disk to disconnect, which sometimes causes a process to get stuck in the dreaded D state and I often end up with corrupt data.Īre there kernel settings I can change to prevent getting into this bad state? I thought echo 1 > /sys/block/sdl/device/queue_depth should prevent concurrent disk operations, but I commonly see around 150 operations in-flight from cat /sys/block/sdl/stat. The only thing that seems to help is decreasing the concurrency of my script writing out files so that disk i/o never gets behind so far that the snowball effect grinds things to a halt. I have even tried a completely different disk controller (which uses the megaraid driver) which did not change anything. I have tried changing to the noop scheduler, setting ncq to 1, and increasing the device timeout to 1 hour. The disks are plugged into a SAS controller that uses the mpt3sas driver (disks show up as scsi devices). I have these seagate disks (ST5000LM000 - note they are SMR) that when I put them under heavy write workloads the i/o utilization goes to 100% and throughput basically goes to zero.

About cache device disk