fea-nux

For Distirbuted ANSYS (DANSYS), when solving a large model over many machines (nodes), the interconnect speed plays an important role in performance.  Regular Ethernet should never be used; Gigabit Ethernet should be considered a minimum.  Gigabit Ethernet (1 GigE) has a maximum speed of 1 gigabit/s or 125 MB/second.  (One can see why Fast Ethernet is not suitable, as it is 1/10 of the speed of Gigabit Ethernet).  1 GigE is very affordable and is commonplace for today’s hardware.  Specialized hardware, such as Infiniband, Quadrics‘ QsNet, or Myricom‘s Myrinet products should be considered, if possible, as they provide even faster speed.  10 Gigabit Ethernet (10 GigE) is also an emerging standard.

At the end of a DANSYS run, latency time and communication speed are printed; an example is shown below:

Latency Time from Master to core    1 =    1.763 microseconds
Latency Time from Master to core    2 =   47.406 microseconds
Latency Time from Master to core    3 =   47.527 microseconds

Communication Speed from Master to core    1 =  1082.66 MB/s
Communication Speed from Master to core    2 =   107.16 MB/s
Communication Speed from Master to core    3 =   107.15 MB/s

In the above sample output, 2 cores on 2 nodes (machines) are used, so that explains the difference in numbers between core 0 (Master) and core 1 as compared with core 0 (Master) and cores 2 & 3.  Because the communication speed is 107 MB/s, one can guess that the interconnect is 1 GigE (which was the case in this example).

Lower latency times and higher communication speeds are better.  With 1 GigE, the communication speed becomes the bottleneck when more than about 8-12 nodes (machines) are used.  Hence, if the user is building a ‘small’ cluster, 1 GigE may be sufficient.  For maximum performance for a ‘larger’ cluster, faster interconnects should be pursued in order to maximize perofrmance.

The ANSYS demo room has additional, useful information on DANSYS performance.  Look for the “HPC for ANSYS 12.0” as well as “Dell and QLogic – Accelerate Performance – Accelerate Business” presentations (free registration required for both).