Category:Parallelization - Revision history

Huebsch: /* Theory */

2024-10-16T06:24:39Z

Theory

← Older revision		Revision as of 06:24, 16 October 2024
Line 11:		Line 11:
	</math>		</math>

	In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.		In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.

	===Communication patterns===		===Communication patterns===

Schlipf: /* MPI setup */

2022-07-26T07:56:18Z

MPI setup

← Older revision		Revision as of 07:56, 26 July 2022
Line 22:		Line 22:
	===MPI setup===		===MPI setup===

	The MPI setup determines the placement of the ranks onto the nodes. VASP assumes the ranks first fill up a node before the next node is occupied. As an example when running with 8 ranks on two nodes, VASP expects rank ~~1–4~~ on node 1 and rank ~~5–8~~ on node 2. If the ranks are placed differently, communication between the nodes occurs for every parallel FFT. Because FFTs are essential to VASP's speed, this deteriorates the performance of the calculation. A manifestation is an increase in computing time when the number of nodes is increased from 1 to 2. If {{TAG\|NCORE}} is not used this issue is less severe but will still reduce the performance.		[[File:good_process_binding.png\|x300px\|All ranks associated with a particular band group are on the same node. All communication for the FFTs is intranode.]]
			[[File:bad_process_binding.png\|x300px\|Ranks of some band groups extend over multiple nodes. Every FFT requires internode communication.]]

			The MPI setup determines the placement of the ranks onto the nodes. VASP assumes the ranks first fill up a node before the next node is occupied. As an example when running with 20 ranks on two nodes, VASP expects rank 1–10 on node 1 and rank 11–20 on node 2. If the ranks are placed differently, communication between the nodes occurs for every parallel FFT. Because FFTs are essential to VASP's speed, this deteriorates the performance of the calculation. A manifestation is an increase in computing time when the number of nodes is increased from 1 to 2. If {{TAG\|NCORE}} is not used this issue is less severe but will still reduce the performance.

	To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.		To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

Schlipf: /* Communication patterns */

2022-07-26T07:48:51Z

Communication patterns

← Older revision		Revision as of 07:48, 26 July 2022
Line 15:		Line 15:
	===Communication patterns===		===Communication patterns===

	The aforementioned parallelization levels directly map onto MPI communicators. Initially, VASP divides all available ranks into {{TAG\|IMAGES}} equally-sized images. Setting {{TAG\|KPAR}} splits each of these images further into groups of '''k''' points. Properties involving all '''k''' points (e.g. density, Fermi energy) require then a communication over these '''k'''-point groups. VASP divides the '''k'''-point group into ~~inter-~~band groups. Communication within the inter-band group occurs to evaluate properties involving two bands (e.g. Hamiltonian, orthonormalization). Therefore it is more frequent than the communication among the '''k'''-point group. The size ~~of the inter-band group is chosen such that~~ {{TAG\|NCORE}} ~~ranks form the intra-band group~~. The ~~intra-~~band group parallelizes the FFTs and triggers the most frequent communications.		[[File:communication.png\|x300px\|VASP divides the available MPI ranks into images, k-point groups, and band groups]]
			[[File:communication2.png\|x300px\|VASP divides the available MPI ranks into images, k-point groups, and band groups]]

			The aforementioned parallelization levels directly map onto MPI communicators. Initially, VASP divides all available ranks into {{TAG\|IMAGES}} equally-sized images. Setting {{TAG\|KPAR}} splits each of these images further into groups of '''k''' points. Properties involving all '''k''' points (e.g. density, Fermi energy) require then a communication over these '''k'''-point groups. VASP divides the '''k'''-point group into band groups of size {{TAG\|NCORE}}. The band group parallelizes the FFTs and triggers the most frequent communications. To evaluate properties involving multiple bands (e.g. Hamiltonian, orthonormalization) communication between the band groups occurs.

	===MPI setup===		===MPI setup===

Schlipf: /* File handling */

2022-07-19T15:21:21Z

File handling

← Older revision		Revision as of 15:21, 19 July 2022
Line 26:		Line 26:
	When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.		When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.
	Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:		Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:
	{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES~~}}, {{TAG\|KIMAGES}}, {{TAG\|FOURORBIT~~}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|~~NCORES_PER_BAND~~}}, {{TAG\|~~NPAR~~}}, {{TAG\|~~NCSHMEM~~}}.		{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|NPAR}}, {{TAG\|NCSHMEM}}, {{TAG\|LUSENCCL}}.
	When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...		When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...
	Any file not present in these subdirectories will be read from the root directory.		Any file not present in these subdirectories will be read from the root directory.
	The output ~~files are~~ files are always written to the subdirectories.		The output files are always written to the subdirectories.
			<!-- The following tags are also read, but the methods are not documented anywhere KIMAGES, FOURORBIT -->

	===Terminology in high-performance computing (HPC)===		===Terminology in high-performance computing (HPC)===

Schlipf: /* Theory */

2022-07-19T15:09:56Z

Theory

← Older revision		Revision as of 15:09, 19 July 2022
Line 22:		Line 22:

	To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.		To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

			===File handling===
			When VASP is started it reads the file {{TAG\|INCAR}} in the root directory.
			Because the MPI setup needs to happen early in the general setup of the calculation the following tags are processed before any of the other settings:
			{{TAG\|NCORE_IN_IMAGE1}}, {{TAG\|IMAGES}}, {{TAG\|KIMAGES}}, {{TAG\|FOURORBIT}}, {{TAG\|KPAR}}, {{TAG\|NCORE}}, {{TAG\|NCORES_PER_BAND}}, {{TAG\|NPAR}}, {{TAG\|NCSHMEM}}.
			When {{TAG\|IMAGES}} are used, subsequently any input given in the root directory is superseded by the same file in subdirectories 01, 02, ...
			Any file not present in these subdirectories will be read from the root directory.
			The output files are files are always written to the subdirectories.

	===Terminology in high-performance computing (HPC)===		===Terminology in high-performance computing (HPC)===

Schlipf: /* Theory */

2022-07-19T08:33:30Z

Theory

← Older revision		Revision as of 08:33, 19 July 2022
Line 12:		Line 12:

	In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.		In addition to the parallelization using MPI, VASP can make use of [[Combining MPI and OpenMP\|OpenMP threading]] and/or [[OpenACC_GPU_port_of_VASP\|OpenACC (for the GPU port)]]. Note that running on multiple OpenMP threads and/or GPUs switches off the {{TAG\|NCORE}} parallelization.

			===Communication patterns===

			The aforementioned parallelization levels directly map onto MPI communicators. Initially, VASP divides all available ranks into {{TAG\|IMAGES}} equally-sized images. Setting {{TAG\|KPAR}} splits each of these images further into groups of '''k''' points. Properties involving all '''k''' points (e.g. density, Fermi energy) require then a communication over these '''k'''-point groups. VASP divides the '''k'''-point group into inter-band groups. Communication within the inter-band group occurs to evaluate properties involving two bands (e.g. Hamiltonian, orthonormalization). Therefore it is more frequent than the communication among the '''k'''-point group. The size of the inter-band group is chosen such that {{TAG\|NCORE}} ranks form the intra-band group. The intra-band group parallelizes the FFTs and triggers the most frequent communications.

	===MPI setup===		===MPI setup===

Huebsch: Huebsch moved page Construction:Parallelization to Category:Parallelization

2022-05-30T08:15:11Z

Huebsch moved page Construction:Parallelization to Category:Parallelization

← Older revision	Revision as of 08:15, 30 May 2022
(No difference)

Schlipf: /* Theory */

2022-05-12T15:27:14Z

Theory

← Older revision		Revision as of 15:27, 12 May 2022
Line 19:		Line 19:
	To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.		To address this issue, please check the setup of the MPI library and the submitted job script. It is usually possible to overwrite the placement by setting environment variables or command-line arguments. When in doubt, contact the HPC administration of your machine to investigate the behavior.

			===Terminology in high-performance computing (HPC)===

	~~===Terminology in high-performance computing (HPC)===~~
	; CPU: The central processing unit of a computer. A CPU may consist of multiple ''cores''. One or more CPUs can be combined with accelerators like ''GPUs'' to form a ''node''. Desktop computers typically contain a single CPU.		; CPU: The central processing unit of a computer. A CPU may consist of multiple ''cores''. One or more CPUs can be combined with accelerators like ''GPUs'' to form a ''node''. Desktop computers typically contain a single CPU.
	; GPU: The graphical processing unit. A GPU is very efficient at matrix and vector operations and may accelerate a program by transferring particularly suitable tasks from the ''CPU'' to the GPU.		; GPU: The graphical processing unit. A GPU is very efficient at matrix and vector operations and may accelerate a program by transferring particularly suitable tasks from the ''CPU'' to the GPU.

Schlipf: /* Theory */

2022-05-12T15:27:00Z

Theory

@@ Line 3: / Line 3: @@
 ==Theory==
 ===Terminology in high-performance computing (HPC)===
@@ Line 18: / Line 34: @@
 <!--; Patches-->
 {{NB|warning|The terminology of nodes, cores, CPUs, threads, etc. is not universal. For instance, some refer to a single core as a CPU, others refer to an entire node as a CPU.}}
 ==How to==

Schlipf: /* Optimizing the parallelization */

2022-05-12T15:26:04Z

Optimizing the parallelization

← Older revision		Revision as of 15:26, 12 May 2022
Line 37:		Line 37:
	===Optimizing the parallelization===		===Optimizing the parallelization===
	The performance of a specific parallelization depends on the system, i.e., the number of ions, the elements, the size of the cell, etc.		The performance of a specific parallelization depends on the system, i.e., the number of ions, the elements, the size of the cell, etc.
	Different algorithms ([[Category:Electronic minimization\|density-functional-theory]], [[:Category:Many-body perturbation theory\|many-body~~–~~perturbation~~–~~theory]], or [[:Category:Molecular dynamics\|molecular-dynamics]] ~~simulation~~) require a separate optimization of the parallel setup.		Different algorithms ([[:Category:Electronic minimization\|density-functional theory]], [[:Category:Many-body perturbation theory\|many-body perturbation theory]], or [[:Category:Molecular dynamics\|molecular dynamics]]) require a separate optimization of the parallel setup.
	To obtain publishable results, many projects require performing many similar calculations, i.e., calculations with similar input and using the same algorithms.		To obtain publishable results, many projects require performing many similar calculations, i.e., calculations with similar input and using the same algorithms.
	Therefore, we recommend optimizing the parallelization to make the most of the available compute time.		Therefore, we recommend optimizing the parallelization to make the most of the available compute time.