Shared memory - Revision history

Huebsch at 09:02, 21 February 2024

2024-02-21T09:02:42Z

← Older revision		Revision as of 09:02, 21 February 2024
Line 64:		Line 64:
	----		----

	[[Category:VASP]][[Category:Installation]][[Category:Performance~~]][[Category:VASP6~~]][[Category:Machine-learned force fields]][[Category:Memory]]		[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:Machine-learned force fields]][[Category:Memory]]

Ftran at 08:27, 20 October 2023

2023-10-20T08:27:14Z

← Older revision		Revision as of 08:27, 20 October 2023
Line 63:		Line 63:

	----		----
	~~[[The_VASP_Manual\|Contents]]~~

	[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:VASP6]][[Category:Machine-learned force fields]][[Category:Memory]]		[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:VASP6]][[Category:Machine-learned force fields]][[Category:Memory]]

Singraber at 13:33, 22 February 2023

2023-02-22T13:33:10Z

← Older revision		Revision as of 13:33, 22 February 2023
Line 26:		Line 26:
	:* Using the MPI-3 shared-memory capabilities (default, any of <tt>-Duse_shmem</tt>, <tt>-Dshmem_bcast_buffer</tt>, <tt>-Dshmem_rproj</tt> but '''not''' <tt>-Dsysv</tt>).		:* Using the MPI-3 shared-memory capabilities (default, any of <tt>-Duse_shmem</tt>, <tt>-Dshmem_bcast_buffer</tt>, <tt>-Dshmem_rproj</tt> but '''not''' <tt>-Dsysv</tt>).
	{{NB\|warning\|Per default, VASP uses MPI-3 calls to allocate and manage shared-memory segments. Unfortunately, we have observed that for some MPI implementations an abnormal termination of the code (''e.g.'' <tt>segfaults</tt> or user initiated abort) does not free these shared-memory segments. This is not a VASP related error. It is caused by the way these shared-memory segments are handled by the operating system and MPI. Without explicit clean-up this leads to a "memory leakage" that persists until the compute node is rebooted. Obviously this is very problematic at high-performance-computing centers. For this reason we do not recommend using shared memory indiscriminately (''i.e.'', without explicit need).\|::}}		{{NB\|warning\|Per default, VASP uses MPI-3 calls to allocate and manage shared-memory segments. Unfortunately, we have observed that for some MPI implementations an abnormal termination of the code (''e.g.'' <tt>segfaults</tt> or user initiated abort) does not free these shared-memory segments. This is not a VASP related error. It is caused by the way these shared-memory segments are handled by the operating system and MPI. Without explicit clean-up this leads to a "memory leakage" that persists until the compute node is rebooted. Obviously this is very problematic at high-performance-computing centers. For this reason we do not recommend using shared memory indiscriminately (''i.e.'', without explicit need).\|::}}
	:* Using <tt>~~icps~~</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>).		:* Using <tt>ipcs</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>).
	{{NB\|tip\|Using <tt>~~icps~~</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>) rarely leads to memory leakage. However, when it does, it is guaranteed to persist until reboot of the node, and no manner of other clean-up will be effective.\|::}}		{{NB\|tip\|Using <tt>ipcs</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>) rarely leads to memory leakage. However, when it does, it is guaranteed to persist until reboot of the node, and no manner of other clean-up will be effective.\|::}}

	::A common problem with the use of <tt>~~icps~~</tt> shared-memory segments and <tt>system-V</tt> semaphores is that the maximum allowed number of semaphores and shared-memory segments, and the maximum allowed size of the latter are system-wide kernel settings. The default settings of many Linux distributions are so strict, ''i.e.'', the allowed number and size of the shared-memory segments is so small, that they are completely unusable for our purposes.		::A common problem with the use of <tt>ipcs</tt> shared-memory segments and <tt>system-V</tt> semaphores is that the maximum allowed number of semaphores and shared-memory segments, and the maximum allowed size of the latter are system-wide kernel settings. The default settings of many Linux distributions are so strict, ''i.e.'', the allowed number and size of the shared-memory segments is so small, that they are completely unusable for our purposes.
	::One can verify the current limits, by:		::One can verify the current limits, by:
	<blockquote>		<blockquote>

Singraber at 11:19, 22 February 2023

2023-02-22T11:19:44Z

← Older revision		Revision as of 11:19, 22 February 2023
Line 14:		Line 14:
	:Use [[Shared memory\|shared-memory]] segments to reduce the storage demands of the [[LREAL\|real-space PAW projectors]].		:Use [[Shared memory\|shared-memory]] segments to reduce the storage demands of the [[LREAL\|real-space PAW projectors]].

	'''-Dsysv'''		'''-Dsysv''' (recommended if possible, see notes below)
	:Use of <tt>ipcs</tt> [[Shared memory\|shared-memory]] segments and <tt>system-V</tt> semaphores '''instead''' of using default MPI-3 shared-memory capabilities (~~recommended if possible,~~ see ~~notes~~ below).		:Use of <tt>ipcs</tt> [[Shared memory\|shared-memory]] segments and <tt>system-V</tt> semaphores '''instead''' of using default MPI-3 shared-memory capabilities (see below).

	In any case the aforementioned [[precompiler options]] have to be accompanied by an additional change to your [[makefile.include]]: The variable <tt>OBJECTS_LIB</tt> needs to contain also the object file <tt>getshmem.o</tt>, e.g., it may look like this:		In any case the aforementioned [[precompiler options]] have to be accompanied by an additional change to your [[makefile.include]]: The variable <tt>OBJECTS_LIB</tt> needs to contain also the object file <tt>getshmem.o</tt>, e.g., it may look like this:

Singraber at 11:14, 22 February 2023

2023-02-22T11:14:21Z

← Older revision		Revision as of 11:14, 22 February 2023
Line 15:		Line 15:

	'''-Dsysv'''		'''-Dsysv'''
	:Use <tt>ipcs</tt> [[Shared memory\|shared-memory]] segments and <tt>system-V</tt> semaphores.		:Use of <tt>ipcs</tt> [[Shared memory\|shared-memory]] segments and <tt>system-V</tt> semaphores '''instead''' of using default MPI-3 shared-memory capabilities (recommended if possible, see notes below).

	:The ~~allocation and handling of shared-memory segments has been implemented in two different ways~~:		In any case the aforementioned [[precompiler options]] have to be accompanied by an additional change to your [[makefile.include]]: The variable <tt>OBJECTS_LIB</tt> needs to contain also the object file <tt>getshmem.o</tt>, e.g., it may look like this:

	:* Using the MPI-3 shared-memory capabilities (default).		OBJECTS_LIB = linpack_double.o getshmem.o
			{{NB\|mind\|If you forget to add <tt>getshmem.o</tt> you may receive errors in the linking stage at the end of the VASP build process, e.g., <code>undefined reference to `getshmem_C'</code>.}}

			The allocation and handling of shared-memory segments has been implemented in two different ways:

			:* Using the MPI-3 shared-memory capabilities (default, any of <tt>-Duse_shmem</tt>, <tt>-Dshmem_bcast_buffer</tt>, <tt>-Dshmem_rproj</tt> but '''not''' <tt>-Dsysv</tt>).
	{{NB\|warning\|Per default, VASP uses MPI-3 calls to allocate and manage shared-memory segments. Unfortunately, we have observed that for some MPI implementations an abnormal termination of the code (''e.g.'' <tt>segfaults</tt> or user initiated abort) does not free these shared-memory segments. This is not a VASP related error. It is caused by the way these shared-memory segments are handled by the operating system and MPI. Without explicit clean-up this leads to a "memory leakage" that persists until the compute node is rebooted. Obviously this is very problematic at high-performance-computing centers. For this reason we do not recommend using shared memory indiscriminately (''i.e.'', without explicit need).\|::}}		{{NB\|warning\|Per default, VASP uses MPI-3 calls to allocate and manage shared-memory segments. Unfortunately, we have observed that for some MPI implementations an abnormal termination of the code (''e.g.'' <tt>segfaults</tt> or user initiated abort) does not free these shared-memory segments. This is not a VASP related error. It is caused by the way these shared-memory segments are handled by the operating system and MPI. Without explicit clean-up this leads to a "memory leakage" that persists until the compute node is rebooted. Obviously this is very problematic at high-performance-computing centers. For this reason we do not recommend using shared memory indiscriminately (''i.e.'', without explicit need).\|::}}
	:* Using <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>).		:* Using <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>).
	{{NB\|tip\|Using <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>) rarely leads to memory leakage. However, when it does, it is guaranteed to persist until reboot of the node, and no manner of other clean-up will be effective.\|::}}		{{NB\|tip\|Using <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores (add [[precompiler options\|precompiler option]] <tt>-Dsysv</tt>) rarely leads to memory leakage. However, when it does, it is guaranteed to persist until reboot of the node, and no manner of other clean-up will be effective.\|::}}

	:A common problem with the use of <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores is that the maximum allowed number of semaphores and shared-memory segments, and the maximum allowed size of the latter are system-wide kernel settings. The default settings of many Linux distributions are so strict, ''i.e.'', the allowed number and size of the shared-memory segments is so small, that they are completely unusable for our purposes.		::A common problem with the use of <tt>icps</tt> shared-memory segments and <tt>system-V</tt> semaphores is that the maximum allowed number of semaphores and shared-memory segments, and the maximum allowed size of the latter are system-wide kernel settings. The default settings of many Linux distributions are so strict, ''i.e.'', the allowed number and size of the shared-memory segments is so small, that they are completely unusable for our purposes.
	:One can verify the current limits, by:		::One can verify the current limits, by:
	<blockquote>		<blockquote>
	<pre>		<pre>
Line 48:		Line 53:
	</pre>		</pre>
	</blockquote>		</blockquote>
	:''i.e.'', on this particular machine the maximum number of semaphores is 32000, the maximum number of shared-memory segments is 4096, and their maximum size is 16Gb.		::''i.e.'', on this particular machine the maximum number of semaphores is 32000, the maximum number of shared-memory segments is 4096, and their maximum size is 16Gb.
	{{NB\|warning\|How to change the maximum number of semaphores and shared-memory segments, and the maximum size of the latter, depends on the particular Linux distribution and generally requires superuser rights. For this reason the use of this implementation of shared memory (<tt>-Dsysv</tt>) is not practical in many situations.\|:}}		{{NB\|warning\|How to change the maximum number of semaphores and shared-memory segments, and the maximum size of the latter, depends on the particular Linux distribution and generally requires superuser rights. For this reason the use of this implementation of shared memory (<tt>-Dsysv</tt>) is not practical in many situations.\|::}}

	== Related articles ==		== Related articles ==

Huebsch at 09:02, 25 May 2022

2022-05-25T09:02:05Z

← Older revision		Revision as of 09:02, 25 May 2022
Line 60:		Line 60:
	[[The_VASP_Manual\|Contents]]		[[The_VASP_Manual\|Contents]]

	[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:VASP6]][[Category:Machine-learned force fields]]		[[Category:VASP]][[Category:Installation]][[Category:Performance]][[Category:VASP6]][[Category:Machine-learned force fields]][[Category:Memory]]

Vaspmaster at 10:54, 8 April 2022

2022-04-08T10:54:02Z

← Older revision		Revision as of 10:54, 8 April 2022
Line 6:		Line 6:

	'''-Duse_shmem'''		'''-Duse_shmem'''
	:Use [[Shared memory\|shared-memory]] segments to reduce the memory demands of [[ALGO\|GW (ALGO = EVGW0, EVGW, QPGW0, and QPGW)]] and [[Machine_learning_force_field_calculations:_Basics#Performance_and_memory_usage\|machine-learned–force-field]] calculations.		:Use [[Shared memory\|shared-memory]] segments to reduce the memory demands of [[ALGO\|GW (ALGO = EVGW0, EVGW, QPGW0, and QPGW)]] and the [[Machine_learning_force_field_calculations:_Basics#Performance_and_memory_usage\|memory demands of machine-learned–force-field]] calculations.

	'''-Dshmem_bcast_buffer'''		'''-Dshmem_bcast_buffer'''

Vaspmaster at 10:52, 8 April 2022

2022-04-08T10:52:20Z

← Older revision		Revision as of 10:52, 8 April 2022
Line 1:		Line 1:
	VASP is mainly [[Parallelization\|parallelized]] using MPI, and as much as practically feasible the computational work and storage demands are distributed over the MPI ranks.		VASP is mainly [[:Category:Parallelization\|parallelized]] using MPI, and as much as practically feasible the computational work and storage demands are distributed over the MPI ranks.
	Unavoidably, however, some data structures are duplicated across all MPI ranks.		Unavoidably, however, some data structures are duplicated across all MPI ranks.
	For some of these data structures, VASP offers the option to reduce the memory consumption by putting them into shared-memory segments. That is segments of memory shared between the MPI ranks that reside on the same compute node and hence have access to the same physical memory.		For some of these data structures, VASP offers the option to reduce the memory consumption by putting them into shared-memory segments. That is segments of memory shared between the MPI ranks that reside on the same compute node and hence have access to the same physical memory.

Vaspmaster at 10:50, 8 April 2022

2022-04-08T10:50:40Z

← Older revision		Revision as of 10:50, 8 April 2022
Line 1:		Line 1:
	VASP is mainly parallelized using MPI, and as much as practically feasible the computational work and storage demands are distributed over the MPI ranks.		VASP is mainly [[Parallelization\|parallelized]] using MPI, and as much as practically feasible the computational work and storage demands are distributed over the MPI ranks.
	Unavoidably, however, some data structures are duplicated across all MPI ranks.		Unavoidably, however, some data structures are duplicated across all MPI ranks.
	For some of these data structures, VASP offers the option to reduce the memory consumption by putting them into shared-memory segments. That is segments of memory shared between the MPI ranks that reside on the same compute node and hence have access to the same physical memory.		For some of these data structures, VASP offers the option to reduce the memory consumption by putting them into shared-memory segments. That is segments of memory shared between the MPI ranks that reside on the same compute node and hence have access to the same physical memory.

Vaspmaster at 08:57, 8 April 2022

2022-04-08T08:57:56Z

← Older revision		Revision as of 08:57, 8 April 2022
Line 49:		Line 49:
	</blockquote>		</blockquote>
	:''i.e.'', on this particular machine the maximum number of semaphores is 32000, the maximum number of shared-memory segments is 4096, and their maximum size is 16Gb.		:''i.e.'', on this particular machine the maximum number of semaphores is 32000, the maximum number of shared-memory segments is 4096, and their maximum size is 16Gb.

	{{NB\|warning\|How to change the maximum number of semaphores and shared-memory segments, and the maximum size of the latter, depends on the particular Linux distribution and generally requires superuser rights. For this reason the use of this implementation of shared memory (<tt>-Dsysv</tt>) is not practical in many situations.\|:}}		{{NB\|warning\|How to change the maximum number of semaphores and shared-memory segments, and the maximum size of the latter, depends on the particular Linux distribution and generally requires superuser rights. For this reason the use of this implementation of shared memory (<tt>-Dsysv</tt>) is not practical in many situations.\|:}}