3.8. qupulse.serialization

This module provides serialization and storage functionality.

Classes:
  • StorageBackend: Abstract representation of a data storage.

  • FilesystemBackend: Implementation of a file system data storage.

  • ZipFileBackend: Like FilesystemBackend but inside a single zip file instead of a directory

  • CachingBackend: A caching decorator for StorageBackends.

  • Serializable: An interface for serializable objects.

  • PulseStorage: High-level management object for loading and storing and transparently (de)serializing serializable objects.

Deprecated Classes:
  • Serializer: Converts Serializables to a serial representation as a string and vice-versa.

Functions:
  • get_default_pulse_registry: Returns the default pulse registry

  • set_default_pulse_registry: Set the default pulse registry

  • new_default_pulse_registry: Reset the default pulse registry with an empty mapping

  • convert_stored_pulse_in_storage: Converts a single Serializable stored using the deprecated Serializer class format into the PulseStorage format.

  • convert_pulses_in_storage: Converts all Serializables stored in a StorageBackend using the deprecated Serializer class format into the PulseStorage format.

Functions

convert_pulses_in_storage(source_storage, ...)

Converts all pulses from the old to the new serialization format.

convert_stored_pulse_in_storage(identifier, ...)

Converts a pulse from the old to the new serialization format.

get_default_pulse_registry()

Returns the current default pulse registry.

new_default_pulse_registry()

Sets a new empty default pulse registry.

set_default_pulse_registry(new_default_registry)

Sets the default pulse registry.

Classes

AnonymousSerializable()

Any object that can be converted into a serialized representation for storage and back which NEVER has an identifier.

CachingBackend(backend)

Adds naive memory caching functionality to another StorageBackend.

DeserializationCallbackFinder()

DictBackend()

DictBackend uses a dictionary to store Serializables in memory.

ExtendedJSONEncoder(*args, **kwargs)

Encodes AnonymousSerializable and sets as lists.

FilesystemBackend([root, create_if_missing])

A StorageBackend implementation based on a regular filesystem.

JSONSerializableDecoder(storage, *args, **kwargs)

JSONDecoder for Serializables.

JSONSerializableEncoder(storage, *args, **kwargs)

JSONEncoder for Serializables.

PulseStorage(storage_backend)

The central storage management for pulses.

Serializable([identifier])

Any object that can be converted into a serialized representation for storage and back.

SerializableMeta(name, bases, dct)

Serializer(storage_backend)

Serializes Serializable objects and stores them persistently.

StorageBackend()

A backend to store data/files in.

ZipFileBackend([root, compression_method])

A StorageBackend implementation based on a single zip file.

class qupulse.serialization.AnonymousSerializable[source]

Bases: object

Any object that can be converted into a serialized representation for storage and back which NEVER has an identifier. This class is used for implicit serialization and does not work necessarily with dicts.

The type information is not saved explicitly but implicitly by the position in the JSON-document.

See also

Serializable

# todo (lumip, 2018-05-30): this does not really have a purpose, especially in the new serialization ecosystem.. we should deprecate and remove it

get_serialization_data() Any[source]

Return all data relevant for serialization as a JSON compatible type that is accepted as constructor argument

Returns

A JSON compatible type that can be used to construct an equal object.

class qupulse.serialization.CachingBackend(backend: qupulse.serialization.StorageBackend)[source]

Bases: qupulse.serialization.StorageBackend

Adds naive memory caching functionality to another StorageBackend.

CachingBackend relies on another StorageBackend to provide real data IO functionality which it extends by caching already opened files in memory for faster subsequent access.

Note that it does not automatically clear the cache at any time and thus will consume increasing amounts of memory over time. Use the clear_cache() method to clear the cache manually.

DEPRECATED (2018-09-20): PulseStorage now already provides chaching around StorageBackends, rendering CachingBackend obsolete.

Creates a new CachingBackend.

Parameters

backend (StorageBackend) – A StorageBackend that provides data IO functionality.

__init__(backend: qupulse.serialization.StorageBackend) None[source]

Creates a new CachingBackend.

Parameters

backend (StorageBackend) – A StorageBackend that provides data IO functionality.

clear_cache() None[source]
delete(identifier: str) None[source]

Deletes data of the given identifier.

Parameters

identifier – identifier of the data to be deleted

Raises

KeyError if there is no data associated with the identifier

exists(identifier: str) bool[source]

Checks if data is stored for the given identifier.

Parameters

identifier (str) – The identifier for which presence of data shall be checked.

Returns

True, if stored data is associated with the given identifier.

get(identifier: str) str[source]

Retrieves the data string with the given identifier.

Parameters

identifier (str) – The identifier of the data to be retrieved.

Returns

A serialized string of the data associated with the given identifier, if present.

Raises

KeyError if no data is associated with the given identifier.

put(identifier: str, data: str, overwrite: bool = False) None[source]

Stores the data string identified by identifier.

Parameters
  • identifier (str) – A unique identifier/name for the data to be stored.

  • data (str) – A serialized string of data to be stored.

  • overwrite (bool) – Set to True, if already existing data shall be overwritten. (default: False)

Raises

FileExistsError if overwrite is False and there already exists data which – is associated with the given identifier.

class qupulse.serialization.DictBackend[source]

Bases: qupulse.serialization.StorageBackend

DictBackend uses a dictionary to store Serializables in memory.

Doing so, it does not provide any persistent storage functionality.

delete(identifier: str) None[source]

Deletes data of the given identifier.

Parameters

identifier – identifier of the data to be deleted

Raises

KeyError if there is no data associated with the identifier

exists(identifier: str) bool[source]

Checks if data is stored for the given identifier.

Parameters

identifier (str) – The identifier for which presence of data shall be checked.

Returns

True, if stored data is associated with the given identifier.

get(identifier: str) str[source]

Retrieves the data string with the given identifier.

Parameters

identifier (str) – The identifier of the data to be retrieved.

Returns

A serialized string of the data associated with the given identifier, if present.

Raises

KeyError if no data is associated with the given identifier.

put(identifier: str, data: str, overwrite: bool = False) None[source]

Stores the data string identified by identifier.

Parameters
  • identifier (str) – A unique identifier/name for the data to be stored.

  • data (str) – A serialized string of data to be stored.

  • overwrite (bool) – Set to True, if already existing data shall be overwritten. (default: False)

Raises

FileExistsError if overwrite is False and there already exists data which – is associated with the given identifier.

property storage: Dict[str, str]
class qupulse.serialization.FilesystemBackend(root: str = '.', create_if_missing: bool = False)[source]

Bases: qupulse.serialization.StorageBackend

A StorageBackend implementation based on a regular filesystem.

Data will be stored in plain text files in a directory. The directory is given in the constructor of this FilesystemBackend. For each data item, a separate file is created an named after the corresponding identifier. If the directory does not exist, it is not created unless the create_if_missing argument is explicitly set to True.

Creates a new FilesystemBackend.

Parameters
  • root – The path of the directory in which all data files are located. (default: “.”, i.e. the current directory)

  • create_if_missing – If False, do not create the specified directory if it does not exist. (default: False)

Raises

NotADirectoryError – if root is not a valid directory path.

__init__(root: str = '.', create_if_missing: bool = False) None[source]

Creates a new FilesystemBackend.

Parameters
  • root – The path of the directory in which all data files are located. (default: “.”, i.e. the current directory)

  • create_if_missing – If False, do not create the specified directory if it does not exist. (default: False)

Raises

NotADirectoryError – if root is not a valid directory path.

delete(identifier)[source]

Deletes data of the given identifier.

Parameters

identifier – identifier of the data to be deleted

Raises

KeyError if there is no data associated with the identifier

exists(identifier: str) bool[source]

Checks if data is stored for the given identifier.

Parameters

identifier (str) – The identifier for which presence of data shall be checked.

Returns

True, if stored data is associated with the given identifier.

get(identifier: str) str[source]

Retrieves the data string with the given identifier.

Parameters

identifier (str) – The identifier of the data to be retrieved.

Returns

A serialized string of the data associated with the given identifier, if present.

Raises

KeyError if no data is associated with the given identifier.

put(identifier: str, data: str, overwrite: bool = False) None[source]

Stores the data string identified by identifier.

Parameters
  • identifier (str) – A unique identifier/name for the data to be stored.

  • data (str) – A serialized string of data to be stored.

  • overwrite (bool) – Set to True, if already existing data shall be overwritten. (default: False)

Raises

FileExistsError if overwrite is False and there already exists data which – is associated with the given identifier.

class qupulse.serialization.PulseStorage(storage_backend: qupulse.serialization.StorageBackend)[source]

Bases: MutableMapping[str, qupulse.serialization.Serializable]

The central storage management for pulses.

Provides a dictionary interface for loading and storing pulses based on any StorageBackend implementation. Takes care of serialization and deserialization of pulses/Serializables in the process. Every Serializable with an identifier will be stored as a separate entity, even if it is nested in another Serializable that is stored. Serializables containing named Serializables will just store a reference to those which will be transparently resolved by PulseStorage during loading.

PulseStorage employs caching, i.e., once a Serializable/pulse is loaded, it will be kept in memory and subsequent fetches will be instantaneous. At the same time, all changes to Serializables known to PulseStorage will immediately be flushed to the storage backend.

Note that it is currently not possible to store a Serializable under a different identifier than the one it holds, i.e. you cannot store a Serializable serializable with identifier ‘foo’ under identifier ‘bar’ by calling pulse_storage[‘bar’] = serializable. This will currently result in a ValueError.

It is also not possible to overwrite a Serializable using the dictionary interface. To explicitly overwrite a Serializable in the storage, use the overwrite method.

PulseStorage can be used as the default pulse registry. All Serializables (and thus pulses) will automatically register themselves with the default pulse registry on construction unless an explicit other registry is provided to them as construction argument. This is intended to prevent accidental duplicate usage of identifiers by failing early. Setting a PulseStorage as pulse default registry also implies that all created Serializables are automatically stored in the storage backend. .. seealso:

PulseStorage.set_to_default_registry
PulseStorage.as_default_registry

Create a PulseStorage instance.

Parameters

storage_backend – The StorageBackend representing the permanent storage of the PulseStorage. Serializables are stored to and read from here.

class StorageEntry(serialization, serializable)

Bases: tuple

Create new instance of StorageEntry(serialization, serializable)

serializable: qupulse.serialization.Serializable

Alias for field number 1

serialization: str

Alias for field number 0

__init__(storage_backend: qupulse.serialization.StorageBackend) None[source]

Create a PulseStorage instance.

Parameters

storage_backend – The StorageBackend representing the permanent storage of the PulseStorage. Serializables are stored to and read from here.

as_default_registry() Any[source]

Returns context manager to use this PulseStorage as the default pulse registry only within a with-statement.

clear() None[source]

Clears the temporary storage.

Does not affect the storage backend.

property contents: Iterable[str]
overwrite(identifier: str, serializable: qupulse.serialization.Serializable) None[source]

Explicitly overwrites a pulse.

Calling this method will overwrite the entity currently stored under the given identifier by the provided serializable. It does _not_ overwrite nested Serializable objects contained in serializable. If you want to overwrite those as well, do that explicitly.

Parameters
  • identifier – The identifier to store serializable under.

  • serializable – The Serializable object to be stored.

set_to_default_registry() None[source]

Promotes this PulseStorage object to be the default pulse registry.

All Serializables (and thus pulses) will automatically register themselves with the default pulse registry on construction unless an explicit other registry is provided to them as construction argument. This is intended to prevent accidental duplicate usage of identifiers by failing early. Setting a PulseStorage as pulse default registry also implies that all created Serializables are automatically stored in the storage backend.

property temporary_storage: Dict[str, qupulse.serialization.StorageEntry]

The in-memory temporary storage.

Contains all Serializables that have been loaded during the lifetime of this PulseStorage object.

class qupulse.serialization.Serializable(identifier: Optional[str] = None)[source]

Bases: object

Any object that can be converted into a serialized representation for storage and back.

Serializable is the interface used by PulseStorage to obtain representations of objects that need to be stored. It essentially provides the methods get_serialization_data, which returns a dictionary which contains all relevant properties of the Serializable object encoded as basic Python types, and deserialize, which is able to reconstruct the object from given such a dictionary.

Additionally, a Serializable object MAY have a unique identifier, which indicates towards the PulseStorage that this object should be stored as a separate data item and accessed by reference instead of possibly embedding it into a containing Serializable’s representation.

All Serializables MUST automatically register themselves with the default pulse registry on construction unless an explicit other registry is provided to them as construction argument. This MUST be implemented by all subclasses of Serializable by calling Serializable._register at some point in their __init__ method. This is intended to prevent accidental duplicate usage of identifiers by failing early.

See also

PulseStorage

Initializes a Serializable.

Parameters

identifier – An optional, non-empty identifier for this Serializable. If set, this Serializable will always be stored as a separate data item and never be embedded.

Raises

ValueError – If identifier is the empty string

__init__(identifier: Optional[str] = None) None[source]

Initializes a Serializable.

Parameters

identifier – An optional, non-empty identifier for this Serializable. If set, this Serializable will always be stored as a separate data item and never be embedded.

Raises

ValueError – If identifier is the empty string

classmethod deserialize(serializer: Optional[qupulse.serialization.Serializer] = None, **kwargs) qupulse.serialization.Serializable[source]

Reconstructs the Serializable object from a dictionary.

Implementation hint: For greater clarity, implementations of this method should be precise in their return value, i.e., give their exact class name, and also replace the kwargs argument by a list of arguments required, i.e., those returned by get_serialization_data. Using old serialization routines, if this Serializable contains complex objects which are itself of type Serializable, their dictionary representations MUST be converted into objects using serializers deserialize() method. This is DEPRECATED behavior. Using the new routines, a serializable is only responsible to decode it’s own dictionary, not those of nested objects (i.e., all incoming arguments are already processed by the serialization routines). For the transition time where both variants are available, implementations of this method should support the old and new routines, using the presence of the serializer argument to differentiate between both. For the new routines, just call this base class function. After the transition period, subclasses likely need not implement deserialize separately anymore at all.

Parameters
  • serializer – DEPRECATED (May 2018). A serializer instance used when deserializing subelements.

  • **kwargs – All relevant properties of the object as keyword arguments. For every (key,value) pair returned by get_serialization_data, the same pair is given as keyword argument as input to this method.

get_serialization_data(serializer: Optional[qupulse.serialization.Serializer] = None) Dict[str, Any][source]

Returns all data relevant for serialization as a dictionary containing only base types.

Implementation hint: In the old serialization routines, if the Serializable contains complex objects which are itself Serializables, a serialized representation for these MUST be obtained by calling the dictify() method of serializer. The reason is that serializer may decide to either return a dictionary to embed or only a reference to the Serializable subelement. This is DEPRECATED behavior as of May 2018. In the new routines, this will happen automatically and every Serializable is only responsible for returning it’s own data and leave nested Serializables in object form.

For the transition time where both implementations are available, implementations of this method should support the old and new routines, using the presence of the serializer argument to differentiate between both. Don’t make use of the implementation in this base class when implementing this method for the old routines.

Parameters

serializer (Serializer) – DEPRECATED (May 2018).A Serializer instance used to serialize complex subelements of this Serializable.

Returns

A dictionary of Python base types (strings, integers, lists/tuples containing these,

etc..) which fully represent the relevant properties of this Serializable for storing and later reconstruction as a Python object.

classmethod get_type_identifier() str[source]
property identifier: Optional[str]

The (optional) identifier of this Serializable. Either a non-empty string or None.

identifier_name = '#identifier'
renamed(new_identifier: str, registry: Optional[MutableMapping[str, qupulse.serialization.Serializable]] = None) qupulse.serialization.Serializable[source]

Returns a copy of the Serializable with its identifier set to new_identifier.

Parameters
  • new_identifier – The identifier of the new copy of this Serializable.

  • registry – The pulse registry the copy of this Serializable will register in. If None, the default pulse registry will be used. Optional.

type_identifier_name = '#type'
class qupulse.serialization.Serializer(storage_backend: qupulse.serialization.StorageBackend)[source]

Bases: object

Serializes Serializable objects and stores them persistently.

DEPRECATED as of May 2018. Serializer will be superseeded by the new serialization routines and PulseStorage class.

Serializer provides methods to enable the conversion of Serializable objects (including nested Serializables) into (nested) dictionaries and serialized JSON-encodings of these and vice-versa. Additionally, it can also store these representations persistently using a StorageBackend instance.

See also

Serializable

Creates a Serializer.

Parameters

storage_backend (StorageBackend) – The StorageBackend all objects will be stored in.

__init__(storage_backend: qupulse.serialization.StorageBackend) None[source]

Creates a Serializer.

Parameters

storage_backend (StorageBackend) – The StorageBackend all objects will be stored in.

deserialize(representation: Union[str, Dict[str, Any]]) qupulse.serialization.Serializable[source]
Loads a stored Serializable object or converts a dictionary representation back to the

corresponding Serializable.

Parameters

representation – A serialization dictionary representing a Serializable object or the identifier of a Serializable object to load from the StorageBackend.

Returns

The Serializable object instantiated from its serialized representation.

See also

Serializable.deserialize

dictify(serializable: qupulse.serialization.Serializable) Union[str, Dict[str, Any]][source]

Converts a Serializable into a dictionary representation.

The Serializable is converted by calling its get_serialization_data() method. If it contains nested Serializables, these are also converted into dictionarys (or references), yielding a single dictionary representation of the outermost Serializable where all nested Serializables are either completely embedded or referenced by identifier.

Parameters

serializable (Serializabe) – The Serializable object to convert.

Returns

A serialization dictionary, i.e., a dictionary of Python base types (strings, integers,

lists/tuples containing these, etc..) which fully represent the relevant properties of the given Serializable for storing and later reconstruction as a Python object. Nested Serializables are either embedded or referenced by identifier.

Raises

Exception if an identifier is assigned twice to different Serializable objects – encountered by this Serializer during the conversion.

See also

Serializable.get_serialization_data

static get_type_identifier(obj: Any) str[source]

Returns a unique type identifier for any object.

Parameters

obj – The object for which to obtain a type identifier.

Returns

The type identifier as a string.

serialize(serializable: qupulse.serialization.Serializable, overwrite=False) None[source]

Serializes and stores a Serializable.

The given Serializable and all nested Serializables that are to be stored separately will be converted into a serial string representation by obtaining their dictionary representation, encoding them as a JSON-string and storing them in the StorageBackend.

If no identifier is given for the Serializable, “main” will be used.

If an identifier is already in use in the StorageBackend, associated data will be replaced.

Parameters

serializable (Serializable) – The Serializable to serialize and store

class qupulse.serialization.StorageBackend[source]

Bases: object

A backend to store data/files in.

Used as an abstraction of file systems/databases for the serializer.

property contents: Iterable[str]

The identifiers of all Serializables currently stored in this StorageBackend.

abstract delete(identifier: str) None[source]

Deletes data of the given identifier.

Parameters

identifier – identifier of the data to be deleted

Raises

KeyError if there is no data associated with the identifier

abstract exists(identifier: str) bool[source]

Checks if data is stored for the given identifier.

Parameters

identifier (str) – The identifier for which presence of data shall be checked.

Returns

True, if stored data is associated with the given identifier.

abstract get(identifier: str) str[source]

Retrieves the data string with the given identifier.

Parameters

identifier (str) – The identifier of the data to be retrieved.

Returns

A serialized string of the data associated with the given identifier, if present.

Raises

KeyError if no data is associated with the given identifier.

list_contents() Iterable[str][source]

Returns a listing of all available identifiers.

DEPRECATED (2018-09-20): Use property contents instead.

Returns

List of all available identifiers.

abstract put(identifier: str, data: str, overwrite: bool = False) None[source]

Stores the data string identified by identifier.

Parameters
  • identifier (str) – A unique identifier/name for the data to be stored.

  • data (str) – A serialized string of data to be stored.

  • overwrite (bool) – Set to True, if already existing data shall be overwritten. (default: False)

Raises

FileExistsError if overwrite is False and there already exists data which – is associated with the given identifier.

class qupulse.serialization.ZipFileBackend(root: str = './storage.zip', compression_method: int = 8)[source]

Bases: qupulse.serialization.StorageBackend

A StorageBackend implementation based on a single zip file.

Data will be stored in plain text files inside a zip file. The zip file is given in the constructor of this FilesystemBackend. For each data item, a separate file is created and named after the corresponding identifier.

ZipFileBackend uses significantly less storage space and is faster on network devices, but takes longer to update because every write causes a complete recompression (it’s not too bad).

Creates a new FilesystemBackend.

Parameters
  • root – The path of the zip file in which all data files are stored. (default: “./storage.zip”, i.e. the current directory)

  • compression_method – The compression method/algorithm used to compress data in the zipfile. Accepts all values handled by the zipfile module (ZIP_STORED, ZIP_DEFLATED, ZIP_BZIP2, ZIP_LZMA). Please refer to the zipfile docs <https://docs.python.org/3/library/zipfile.html#zipfile.ZIP_STORED> for more information. (default: zipfile.ZIP_DEFLATED)

Raises

NotADirectoryError if root is not a valid path.

__init__(root: str = './storage.zip', compression_method: int = 8) None[source]

Creates a new FilesystemBackend.

Parameters
  • root – The path of the zip file in which all data files are stored. (default: “./storage.zip”, i.e. the current directory)

  • compression_method – The compression method/algorithm used to compress data in the zipfile. Accepts all values handled by the zipfile module (ZIP_STORED, ZIP_DEFLATED, ZIP_BZIP2, ZIP_LZMA). Please refer to the zipfile docs <https://docs.python.org/3/library/zipfile.html#zipfile.ZIP_STORED> for more information. (default: zipfile.ZIP_DEFLATED)

Raises

NotADirectoryError if root is not a valid path.

delete(identifier: str) None[source]

Deletes data of the given identifier.

Parameters

identifier – identifier of the data to be deleted

Raises

KeyError if there is no data associated with the identifier

exists(identifier: str) bool[source]

Checks if data is stored for the given identifier.

Parameters

identifier (str) – The identifier for which presence of data shall be checked.

Returns

True, if stored data is associated with the given identifier.

get(identifier: str) str[source]

Retrieves the data string with the given identifier.

Parameters

identifier (str) – The identifier of the data to be retrieved.

Returns

A serialized string of the data associated with the given identifier, if present.

Raises

KeyError if no data is associated with the given identifier.

put(identifier: str, data: str, overwrite: bool = False) None[source]

Stores the data string identified by identifier.

Parameters
  • identifier (str) – A unique identifier/name for the data to be stored.

  • data (str) – A serialized string of data to be stored.

  • overwrite (bool) – Set to True, if already existing data shall be overwritten. (default: False)

Raises

FileExistsError if overwrite is False and there already exists data which – is associated with the given identifier.

qupulse.serialization.convert_pulses_in_storage(source_storage: qupulse.serialization.StorageBackend, dest_storage: qupulse.serialization.StorageBackend) None[source]

Converts all pulses from the old to the new serialization format.

All pulses in a given source storage are completely (including subpulses) converted from the old serialization format to the new serialization format and written to a given destination storage.

Parameters
  • source_storage (StorageBackend) – A StorageBackend containing pulses in the old serialization format.

  • dest_storage (StorageBackend) – A StorageBackend the converted pulses will be written to in the new serialization format.

Raises

ValueError – if the dest_storage StorageBackend contains identifiers also assigned in source_storage.

qupulse.serialization.convert_stored_pulse_in_storage(identifier: str, source_storage: qupulse.serialization.StorageBackend, dest_storage: qupulse.serialization.StorageBackend) None[source]

Converts a pulse from the old to the new serialization format.

The pulse with the given identifier is completely (including subpulses) converted from the old serialization format read from a given source storage to the new serialization format and written to a given destination storage.

Parameters
  • identifier (str) – The identifier of the pulse to convert.

  • source_storage (StorageBackend) – A StorageBackend containing the pulse identified by the identifier argument in the old serialization format.

  • dest_storage (StorageBackend) – A StorageBackend the converted pulse will be written to in the new serialization format.

Raises

ValueError – if the dest_storage StorageBackend contains identifiers also assigned in source_storage.

qupulse.serialization.get_default_pulse_registry() Optional[MutableMapping[str, qupulse.serialization.Serializable]][source]

Returns the current default pulse registry.

qupulse.serialization.new_default_pulse_registry() None[source]

Sets a new empty default pulse registry.

The new registry is a newly created weakref.WeakValueDictionry().

qupulse.serialization.set_default_pulse_registry(new_default_registry: Optional[MutableMapping[str, qupulse.serialization.Serializable]]) None[source]

Sets the default pulse registry.

Parameters

new_default_registry – Any PulseRegistryType object (i.e., mutable mapping) which will become the new default pulse registry.