On large zpools ZFS ARC memory occupies whole free ram on the machine and it starts swapping. It looks like that something is wrong and it needs some tuning.
Good results are archived after founding ZFS sysctl tunable vfs.zfs.arc_free_target to define how much memory should be left free from ZFS ARC for other processes. It counts in 4096 bit blocks. You can check the default value using command „sysctl vfs.zfs.arc_free_target“.
So my recommendation for low memory systems is:
- vfs.zfs.arc_free_target:32768 (128 MB)
For normal servers:
- vfs.zfs.arc_free_target:524288 (2 GB)