logging_handlers

class pyrosetta.distributed.cluster.logging_handlers.HandlerMixin

Bases: object

Logging handler mixin class for acquiring and releasing a thread lock.

static lock(func: L) → L

A decorator for methods requiring acquire and release of a thread lock.

_locked() → Generator[Any, Any, Any]

A context manager for acquiring and releasing a thread lock.

class pyrosetta.distributed.cluster.logging_handlers.MsgpackHmacSocketHandler(host: str, port: int)

Bases: SocketHandler, HandlerMixin

Subclass of logging.handlers.SocketHandler using MessagePack and hash-based message authentication codes (HMAC).

_supported_types = (<class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>, <class 'NoneType'>, <class 'bytes'>, <class 'bytearray'>)

__init__(host: str, port: int) → None

Initializes the handler with a specific host address and port.

When the attribute closeOnError is set to True - if a socket error occurs, the socket is silently closed and then reopened on the next logging call.

set_masked_key(task_id: str, masked_key: bytes) → None

Set a task ID and HMAC key into the cache.

pop_masked_key(task_id: str) → None

Pop a task ID and HMAC key from the cache.

clear_masked_keys() → None

Clear all task IDs and HMAC keys from the cache.

zeroize(buffer: Optional[bytearray]) → None

Zeroize a bytearray in memory.

close() → None

Close the handler.

sanitize_record_arg(arg: Any) → Any

Sanitize a single element of log record args for MessagePack.

sanitize_record_args(args: Any) → Any

Sanitize log record args for MessagePack.

makePickle(record: LogRecord) → bytes

Compress a logging record with MessagePack and a hash-based message authentication code (HMAC).

_at_fork_reinit()

_locked() → Generator[Any, Any, Any]

A context manager for acquiring and releasing a thread lock.

acquire()

Acquire the I/O thread lock.

addFilter(filter)

Add the specified filter to this handler.

createLock()

Acquire a thread lock for serializing access to the underlying I/O.

createSocket()

Try to create a socket, using an exponential backoff with a max retry time. Thanks to Robert Olson for the original patch (SF #815911) which has been slightly refactored.

emit(record)

Emit a record.

Pickles the record and writes it to the socket in binary format. If there is an error with the socket, silently drop the packet. If there was a problem with the socket, re-establishes the socket.

filter(record)

Determine if a record is loggable by consulting all the filters.

The default is to allow the record to be logged; any filter can veto this and the record is then dropped. Returns a zero value if a record is to be dropped, else non-zero.

Changed in version 3.2: Allow filters to be just callables.

flush()

Ensure all logging output has been flushed.

This version does nothing and is intended to be implemented by subclasses.

format(record)

Format the specified record.

If a formatter is set, use it. Otherwise, use the default formatter for the module.

get_name()

handle(record)

Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler. Wrap the actual emission of the record with acquisition/release of the I/O thread lock. Returns whether the filter passed the record for emission.

handleError(record)

Handle an error during logging.

An error has occurred during logging. Most likely cause - connection lost. Close the socket so that we can retry on the next event.

static lock(func: L) → L

A decorator for methods requiring acquire and release of a thread lock.

makeSocket(timeout=1)

A factory method which allows subclasses to define the precise type of socket they want.

property name

release()

Release the I/O thread lock.

removeFilter(filter)

Remove the specified filter from this handler.

send(s)

Send a pickled string to the socket.

This function allows for partial sends which can happen when the network is busy.

setFormatter(fmt)

Set the formatter for this handler.

setLevel(level)

Set the logging level of this handler. level must be an int or a str.

set_name(name)

class pyrosetta.distributed.cluster.logging_handlers.MultiSocketHandler(logging_level: Union[str, int] = 0, maxsize: int = 128)

Bases: Handler, HandlerMixin

Cache mutable dask worker logger handlers up to a maximum size, pruning least recently used (LRU) dask worker loggers first.

__init__(logging_level: Union[str, int] = 0, maxsize: int = 128) → None

Initializes the instance - basically setting the formatter to None and the filter list to empty.

set_masked_key(socket_listener_address: Tuple[str, int], task_id: str, masked_key: bytes) → None

Set a masked key to handler cache.

pop_masked_key(socket_listener_address: Tuple[str, int], task_id: str) → None

Pop a masked key a handler cache.

setup_handler(host: str, port: int) → MsgpackHmacSocketHandler

Setup a MsgpackHmacSocketHandler instance.

get(host: str, port: int) → Tuple[Tuple[str, int], MsgpackHmacSocketHandler]

Set a key as most recently used, and return the key and value from the cache.

emit(record: LogRecord) → None

Logging handler custom emit method override.

maybe_prune() → None

Prune the least recently used (LRU) items within the maximum size of the cache.

purge_address(key: Tuple[str, int]) → None

Close and remove an item from the cache.

purge_all() → None

Close and remove all items from the cache.

close() → None

Logging handler custom close method override.

_at_fork_reinit()

_locked() → Generator[Any, Any, Any]

A context manager for acquiring and releasing a thread lock.

acquire()

Acquire the I/O thread lock.

addFilter(filter)

Add the specified filter to this handler.

createLock()

Acquire a thread lock for serializing access to the underlying I/O.

filter(record)

Determine if a record is loggable by consulting all the filters.

The default is to allow the record to be logged; any filter can veto this and the record is then dropped. Returns a zero value if a record is to be dropped, else non-zero.

Changed in version 3.2: Allow filters to be just callables.

flush()

Ensure all logging output has been flushed.

This version does nothing and is intended to be implemented by subclasses.

format(record)

Format the specified record.

If a formatter is set, use it. Otherwise, use the default formatter for the module.

get_name()

handle(record)

Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler. Wrap the actual emission of the record with acquisition/release of the I/O thread lock. Returns whether the filter passed the record for emission.

handleError(record)

Handle errors which occur during an emit() call.

This method should be called from handlers when an exception is encountered during an emit() call. If raiseExceptions is false, exceptions get silently ignored. This is what is mostly wanted for a logging system - most users will not care about errors in the logging system, they are more interested in application errors. You could, however, replace this with a custom handler if you wish. The record which was being processed is passed in to this method.

static lock(func: L) → L

A decorator for methods requiring acquire and release of a thread lock.

property name

release()

Release the I/O thread lock.

removeFilter(filter)

Remove the specified filter from this handler.

setFormatter(fmt)

Set the formatter for this handler.

setLevel(level)

Set the logging level of this handler. level must be an int or a str.

set_name(name)

pyrosetta.distributed.cluster.logging_handlers.get_stdout_handler(logging_level: Union[str, int] = 0) → Handler

Get a logging stream handler to sys.stdout for root logger records from dask workers.