Using network shares
When using ANSYS — especially the direct sparse solver (EQSLV,SPARSE or EQSLV,DSPARSE) — one should not solve on a network share if using only Fast Ethernet (100 Mbit/s ~ 12 MB/s) or Gigabit Ethernet (1000 Mbit/s ~ 120 MB/s). It is common on Linux to have the /home directory shared, and this can be verified with the df command; on Windows, one may have the “home” directory also on a network mount.
Some ANSYS scratch solver files can be relatively large, depending on the element types used and the size of the model; these include the .emat and .esav files. The sparse direct solver also writes large files — this is due to the fact that the inverse of the sparse matrix can have many non-zero terms, so the factorized matrix can be many gigabytes in size for larger models. The factorized matrix needs to be written/read 3 times for out-of-core solutions, so a lot of disk I/O may occur. If the network interconnect is only Fast Ethernet (~12 MB/s), this is much slower than typical hard drive speeds that can range from 80 to 200 MB/s.
The situation would become even worse with Distributed ANSYS (DANSYS). If there are 4 processes being solved on different nodes in a cluster, all working from an NFS share, not only will there be a lot of communication between the master and spawned processes, but each process will be reading/writing data to the network share. This would result it a lot of network traffic, thus reducing performance considerably.
For situations where a lot of disk I/O may be expected, change the working directory to local disk. On Windows, this can be as simple as copying your working folder to your C: or D: drive. On Linux, the system administrator may need to be contacted to give you read/write access to a local partition. Doing so should improve performance greatly. Solver statistics in the .BCS file (sparse solver) or .DSP file (distributed sparse solver) can be used to see the change in the effective I/O rate and improvement in the solver times before and after using a local partition.
I’m testing a graphic blade from hp right now (ProLiant xw2x220c Blade Workstation). The idea is to get rid of all the workstations and work remotely on the graphic blades.
I’m pretty happy with the system: graphic performance is great and the computation performance is neat. There is only one big downside: HP blessed the blade with a painfully slow hard drive.
The idea was then to use a faster hard drive on another node which is connected by Gigabit Ethernet. Not the perfect solution, I know, but worth a try. But it seems like Ansys can’t handle that, it stops the solution with a vacuous message. However, I haven’t tried hard to get it running but sooner or later I’ll give it another shot.
Hi Fabian,
You may want to try benchmarking the I/O speed from the Blade Workstation to the other computer compared with I/O speed locally just to get a quantitative difference in speeds. Gigabit Ethernet would only give about 110 MB/s or so max.
If you have ANSYS stopping and you’re using Linux, check the NFS settings on the file server – you may want to try “sync” instead of “async”. Asynchronous mode is much faster, but I have come across situations where ANSYS tries to read/write data that hasn’t actually be read/written on the file server, thus reading in an I/O error. (If ANSYS stops with no error message at all as you implied in your comment, then this setting may not be the cause…)