ML LOGFILE - Revision history

Schlipf at 13:35, 7 March 2025

2025-03-07T13:35:44Z

← Older revision		Revision as of 13:35, 7 March 2025
Line 261:		Line 261:
	----		----

	[[Category:Files]][[Category:Machine-learned force fields]][[Category:Output ~~Files~~]]		[[Category:Files]][[Category:Machine-learned force fields]][[Category:Output files]]

Singraber at 15:01, 19 October 2023

2023-10-19T15:01:14Z

← Older revision		Revision as of 15:01, 19 October 2023
Line 15:		Line 15:
	********************************************************************************************************************************************		********************************************************************************************************************************************

	The actual composition of log sections may depend on the machine learning mode of operation (see {{TAG\|~~ML_ISTART~~}}). Usually, in the beginning there will be a couple of sections describing the estimated memory consumption, machine learning settings and preexisting data. Then follows the main loop, which is split into a header and the actual loop body containing data describing the learning progress ({{~~TAG~~\|~~ML_ISTART~~}} ~~= 0, 1~~) or prediction ({{~~TAG~~\|~~ML_ISTART~~}} ~~= 2~~). Finally, there may be sections about actual memory consumption and timing statistics. The following chapters describe the contents of the log file sections in more detail:		The actual composition of log sections may depend on the machine learning mode of operation (see {{TAG\|ML_MODE}}). Usually, in the beginning there will be a couple of sections describing the estimated memory consumption, machine learning settings and preexisting data. Then follows the main loop, which is split into a header and the actual loop body containing data describing the learning progress ({{TAGO\|ML_MODE\|train}}) or prediction ({{TAGO\|ML_MODE\|run}}). Finally, there may be sections about actual memory consumption and timing statistics. The following chapters describe the contents of the log file sections in more detail:

	=== Memory consumption estimation ===		=== Memory consumption estimation ===

	This is usually the first section of the {{TAG\|ML_LOGFILE}} and contains an '''estimation''' of memory requirements based on {{VASP}} files read on startup. In the simplest case ({{~~TAG~~\|~~ML_ISTART~~}} ~~= 0~~) it depends on the settings in the {{FILE\|INCAR}} and {{FILE\|POSCAR}} file. For example, the expected memory consumption may vary with the number of elements present in the {{FILE\|POSCAR}} file. Various {{FILE\|INCAR}} tags also influence the memory demand, e.g. {{TAG\|ML_MB}} or {{TAG\|ML_MRB2}}. A continuation or prediction run, i.e. {{~~TAG~~\|~~ML_ISTART~~}} ~~= 1, 2~~, may also take settings from the files {{FILE\|ML_AB}} or {{FILE\|ML_FF}} into account.		This is usually the first section of the {{TAG\|ML_LOGFILE}} and contains an '''estimation''' of memory requirements based on {{VASP}} files read on startup. In the simplest case ({{TAGO\|ML_MODE\|train}}) it depends on the settings in the {{FILE\|INCAR}} and {{FILE\|POSCAR}} file. For example, the expected memory consumption may vary with the number of elements present in the {{FILE\|POSCAR}} file. Various {{FILE\|INCAR}} tags also influence the memory demand, e.g. {{TAG\|ML_MB}} or {{TAG\|ML_MRB2}}. A continuation or prediction run, i.e., {{TAGO\|ML_MODE\|train}} with {{FILE\|ML_AB}} present or {{TAGO\|ML_MODE\|run}}, may also take settings from the files {{FILE\|ML_AB}} or {{FILE\|ML_FF}} into account.

	* MEMORY INFORMATION ***********************************************************************************************************************		* MEMORY INFORMATION ***********************************************************************************************************************
Line 80:		Line 80:
	General settings		General settings
	--------------------------------------------------------------------------------------------------------------------------------------------		--------------------------------------------------------------------------------------------------------------------------------------------
	Machine learning operation mode : 0 (I) ~~ML_ISTART~~		Machine learning operation mode in strings (supertag) : REFIT (I) ML_MODE
			Machine learning operation mode : 4 ML_ISTART
			Precontraction of weights on Kernel for fast execution (ML_ISTART=2 only), but no error estimation : T ML_LFAST
			Controls the verbosity of the output at each MD step when machine learning is used : 1 ML_OUTPUT_MODE
			Sets the output frequency at various places for ML_ISTART=2 : 1 ML_OUTBLOCK


	Descriptor settings		Descriptor settings
	--------------------------------------------------------------------------------------------------------------------------------------------		--------------------------------------------------------------------------------------------------------------------------------------------
Line 101:		Line 105:
	=== Existing ab initio data ===		=== Existing ab initio data ===

	This section will appear in continuation runs (e.g. ~~<code>~~{{~~TAG~~\|~~ML_ISTART~~}} ~~= 1</code>~~) and summarizes the ab initio data found in the {{FILE\|ML_AB}} file.		This section will appear in continuation runs (e.g. {{TAGO\|ML_MODE\|train}} with existing {{FILE\|ML_AB}}) and summarizes the ab initio data found in the {{FILE\|ML_AB}} file.

	* AVAILABLE AB INITIO DATA *****************************************************************************************************************		* AVAILABLE AB INITIO DATA *****************************************************************************************************************
Line 115:		Line 119:
	=== Main loop ===		=== Main loop ===

	The central part of the {{FILE\|ML_LOGFILE}} is the main loop: depending on the machine learning mode {{TAG\|~~ML_ISTART~~}} it contains data collected over all the time steps (or other iterative schemes) along the {{VASP}} run. The main loop layout is carefully designed to minimize the file size while at the same being self-descriptive. To achieve this, it is split into two parts: the description blocks in the main loop header explain the available data and present its arrangement in lines and columns. Then, the main loop body contains the actual data (mostly raw numbers) in the previously defined layout. The separation of data and its description avoids unnecessary repetition and simplifies [[#Post-processing usage\|post-processing]].		The central part of the {{FILE\|ML_LOGFILE}} is the main loop: depending on the machine learning mode {{TAG\|ML_MODE}} it contains data collected over all the time steps (or other iterative schemes) along the {{VASP}} run. The main loop layout is carefully designed to minimize the file size while at the same being self-descriptive. To achieve this, it is split into two parts: the description blocks in the main loop header explain the available data and present its arrangement in lines and columns. Then, the main loop body contains the actual data (mostly raw numbers) in the previously defined layout. The separation of data and its description avoids unnecessary repetition and simplifies [[#Post-processing usage\|post-processing]].

	==== Header ====		==== Header ====

Singraber: /* File layout */

2023-01-16T08:34:35Z

File layout

← Older revision		Revision as of 08:34, 16 January 2023
Line 55:		Line 55:
	{{Template:MLFF_memorgy_usage_table}}		{{Template:MLFF_memorgy_usage_table}}

	The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also <code>Descriptors</code> and <code>Prediction</code> will drop in memory, but only slightly since only minor parts of these are distributed). The parts marked with an ''x'' for shared memory show also a significant decrease of memory usage if the code is compiled for shared memory (precompiler option ''use_shmem''). The risks of shared memory usage are explained [[Machine learning force field calculations: Basics#Shared memory with MPI\|here]].		The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also <code>Descriptors</code> and <code>Prediction</code> will drop in memory, but only slightly since only minor parts of these are distributed). The parts marked with an ''x'' for shared memory show also a significant decrease of memory usage if the code is compiled for shared memory (precompiler option ''use_shmem''). The risks of shared memory usage are explained [[Machine learning force field calculations: Basics#Shared memory with MPI\|here]]. {{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}

	{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}

	=== Machine learning setup ===		=== Machine learning setup ===

Karsai at 13:26, 8 April 2022

2022-04-08T13:26:37Z

← Older revision		Revision as of 13:26, 8 April 2022
Line 259:		Line 259:
	----		----

	[[Category:Files]][[Category:Machine ~~Learning]][[Category:Machine Learned Force Fields~~]][[Category:Output Files]]		[[Category:Files]][[Category:Machine-learned force fields]][[Category:Output Files]]

Karsai at 07:33, 7 April 2022

2022-04-07T07:33:02Z

← Older revision		Revision as of 07:33, 7 April 2022
Line 1:		Line 1:
			{{DISPLAYTITLE:ML_LOGFILE}}
	Every {{VASP}} run with activated machine learning ({{FILE\|INCAR}} contains {{TAG\|ML_LMLFF}} = .TRUE.) will generate a file called {{TAG\|ML_LOGFILE}}. In this log file a summary of settings and the development of quantities related to machine learning are presented in a compact, yet human-readable and post-processing friendly way. It complements the usual ab initio log output in the {{FILE\|OUTCAR}} and {{FILE\|OSZICAR}} files for machine learning VASP runs.		Every {{VASP}} run with activated machine learning ({{FILE\|INCAR}} contains {{TAG\|ML_LMLFF}} = .TRUE.) will generate a file called {{TAG\|ML_LOGFILE}}. In this log file a summary of settings and the development of quantities related to machine learning are presented in a compact, yet human-readable and post-processing friendly way. It complements the usual ab initio log output in the {{FILE\|OUTCAR}} and {{FILE\|OSZICAR}} files for machine learning VASP runs.
	<!-- This file contains the main output for the machine learning force field method.		<!-- This file contains the main output for the machine learning force field method.
Line 258:		Line 259:
	----		----

	[[Category:Files]][[Category:Machine Learning]][[Category:Machine Learned Force Fields]][[Category:Output Files~~]][[Category:Alpha~~]]		[[Category:Files]][[Category:Machine Learning]][[Category:Machine Learned Force Fields]][[Category:Output Files]]

Karsai: /* Memory consumption estimation */

2022-01-07T12:59:01Z

Memory consumption estimation

← Older revision		Revision as of 12:59, 7 January 2022
Line 54:		Line 54:
	{{Template:MLFF_memorgy_usage_table}}		{{Template:MLFF_memorgy_usage_table}}

	The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also <code>Descriptors</code> and <code>Prediction</code> will drop in memory, but only slightly since only ~~very small~~ parts of these are distributed).		The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also <code>Descriptors</code> and <code>Prediction</code> will drop in memory, but only slightly since only minor parts of these are distributed). The parts marked with an ''x'' for shared memory show also a significant decrease of memory usage if the code is compiled for shared memory (precompiler option ''use_shmem''). The risks of shared memory usage are explained [[Machine learning force field calculations: Basics#Shared memory with MPI\|here]].

	{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}		{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}

Karsai: /* Memory consumption estimation */

2022-01-07T12:29:55Z

Memory consumption estimation

← Older revision		Revision as of 12:29, 7 January 2022
Line 54:		Line 54:
	{{Template:MLFF_memorgy_usage_table}}		{{Template:MLFF_memorgy_usage_table}}

	The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also ''Descriptors'' and ''Prediction'' will drop in memory, but only slightly since only very small parts of these are distributed).		The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also <code>Descriptors</code> and <code>Prediction</code> will drop in memory, but only slightly since only very small parts of these are distributed).

	{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}		{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}

Karsai: /* Memory consumption estimation */

2022-01-07T12:28:55Z

Memory consumption estimation

← Older revision		Revision as of 12:28, 7 January 2022
Line 51:		Line 51:
	While the individual items in the above listing are of rather technical nature the most important number is given in the last line: <code>Total memory consumption</code> approximates the peak memory usage during this {{VASP}} run. However, since not all memory is always allocated at the same time the actual consumption may vary over time.		While the individual items in the above listing are of rather technical nature the most important number is given in the last line: <code>Total memory consumption</code> approximates the peak memory usage during this {{VASP}} run. However, since not all memory is always allocated at the same time the actual consumption may vary over time.

	The following part summarizes ~~how~~ each ~~part might scale in memory with the number of CPUs~~:		The following part summarizes which kind of parallelization each parts employ:
	{{Template:MLFF_memorgy_usage_table}}		{{Template:MLFF_memorgy_usage_table}}

			The parts marked with an ''x'' for scaLAPACK contain block-cyclic distributed arrays that usually scale almost perfectly with the number of employed processors. The user will see that by increasing the number of processors the amount of memory needed will significantly drop for that part (we not that also ''Descriptors'' and ''Prediction'' will drop in memory, but only slightly since only very small parts of these are distributed).

	{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}		{{NB\|mind\|This is only an estimate, the actual memory requirement may be even higher. Moreover, this is only the usage for the machine learning part of {{VASP}} which in a training run adds up to the memory of the ab initio part.}}

Karsai: /* Memory consumption estimation */

2022-01-07T12:15:35Z

Memory consumption estimation

← Older revision		Revision as of 12:15, 7 January 2022
Line 38:		Line 38:
	Temporary allocations for sparsification : 460.9		Temporary allocations for sparsification : 460.9
	\|		\|
	\|-- SVD matrices : 28.0		\|-- SVD matrices : 460.7

	Other temporary allocations : 15.5		Other temporary allocations : 15.5

Karsai: /* Memory consumption estimation */

2022-01-07T12:14:42Z

Memory consumption estimation

← Older revision		Revision as of 12:14, 7 January 2022
Line 51:		Line 51:
	While the individual items in the above listing are of rather technical nature the most important number is given in the last line: <code>Total memory consumption</code> approximates the peak memory usage during this {{VASP}} run. However, since not all memory is always allocated at the same time the actual consumption may vary over time.		While the individual items in the above listing are of rather technical nature the most important number is given in the last line: <code>Total memory consumption</code> approximates the peak memory usage during this {{VASP}} run. However, since not all memory is always allocated at the same time the actual consumption may vary over time.

	~~A table summarizing the behaviour of~~ each ~~line~~ in the ~~output~~ of ~~the memory is given here~~:		The following part summarizes how each part might scale in memory with the number of CPUs:
	{{Template:MLFF_memorgy_usage_table}}		{{Template:MLFF_memorgy_usage_table}}