Today i decided to find out why the write speed is slow on my Ceph cluster. Read speed is good. I noticed that i have high utilization even on slow write speeds. I cant get higher write speeds than 30MB/s. Then the disk utilization is close to 100%. I have put block db and write ahead log on SSDs. I expected that to make the write speeds much better. I see now that the SSDs are little used. Most writes go straight to the HDD. The HDDs can write 200MB/s and i have 10gbps network between the nodes. The problem is that Ceph always do synchronous writes. That means the write call must not return before the data is written to stable storage. That makes both HDDs and SSDs much slower. To get faster writes i need SSDs with power loss protection. SSDs with PLP can say that data is written to storage as soon as the data is in the SSDs cache, because the SSD knows it have enough power to write the data to storage even if there is a power loss.
The write speed is much better on my Windows cluster. Storage spaces direct writes data first to the cache SSD and says that data is written to stable storage. I have SSDs with PLP on two nodes. On one node the SSD dont have PLP so i turn on write cache in device manager every time i restart the node. Storage spaces direct turns it off every restart.
