aphylogeo.multiProcessor

class aphylogeo.multiProcessor.Multi(args, function)

Bases: object

Class that harnesses the power of “multiprocess” and wraps it, for ease of use, as a single callable line. It is mainly aimed at very many large tasks that could be run on supercomputers with an ridiculous amount of RAM.

executeOnce(arg)

Executes a method as a single process.

This method runs a specified function with the given argument and stores the result in self.resultList.

Parameters:

arg (list) – The list of arguments given to the method.

Returns:

The return value of the executed method is passed to self.resultList.

Return type:

None

processingLargeData()

Method for executing multiprocessing on tasks that require a large amount of individual memory. It first runs a single process and then starts as many child processes as the available RAM permits, launching new ones as RAM is freed.

Variables:

pProcess

Represents a single child process.

Returns:

The multiprocess-friendly list that is updated by each child process.

Return type:

list

Errors:

If another application reduces the available RAM mid-execution, the multiprocessing framework outputs “Killed” and terminates the child process.

buttler(memBloc)

Ran as a child process, the buttler will constantly run other methods forever.

In this case, it:

• Updates the memory capacity.

• Prints updates to the terminal

It exists so not to bottleneck the main thread.

Uses timers to execute it’s methods because time.sleep() it processor hungry if constantly called

memUpdate()

Method that sets the baseline for memory calculation + output some information to the terminal All memory values are in bytes

This method is ran from the buttler() and updates every second

Variables:

memBufferfloat

Percentage of bytes to subtract from the available RAM for safety purposes.

memfloat

Amount of bytes.

terminalUpdate(memBlock)

Method that constantly updates the user about the currently run tasks

This method is ran from the buttler() and updates every 0.1 second

executeSmall(arg)

The method executed by processingSmallData

Returns:

Nothing, but the return value of the executed method is passed to a global multiprocessing-friendly list

processingSmallData()

Method for executing mutliprocess on tasks that demand little to no memory. Will immedialty start all the child processes. The packaging causes some marginal time lost; Only use for methods that take at least a second to run : below that, a for loop is likely much faster

Variables:

pProcess

Represents a single child process.

Returns:

The multiprocess-friendly list, that was updated by each child

Errors:

If other application reduce the avalable RAM mid-execution, Multiprocess outputs “Killed” and kills the child.