The exponential_retry Module

cloudfusion.util.exponential_retry.retry(ExceptionToCheck, tries=20, delay=0.1, backoff=2)

Retry calling the decorated function using an exponential backoff. The decorated method’s object may define the _handle_error method. The parameters for _handle_error are the exception object, a formatted stack trace in the form of a string, the wrapped methods’ name, the number of tries remaining, and finally the original parameters. If _handle_error returns True, retrying is aborted.

Parameters:
  • ExceptionToCheck (Exception or tuple) – the exception to check. may be a tuple of exceptions to check
  • tries (int) – number of times to try (not retry) before giving up
  • delay (int) – initial delay between retries in seconds
  • backoff (int) – backoff multiplier e.g. value of 2 will double the delay each retry

The pickle_methods Module

This module can be imported to allow pickling of methods.

cloudfusion.util.pickle_methods._pickle_method(method)
cloudfusion.util.pickle_methods._unpickle_method(func_name, obj, cls)

The file_util Module

Created on Sep 26, 2014

@author: joe

cloudfusion.util.file_util.get_file_size_in_bytes(file)

file may be the path to a file or a fileobject

cloudfusion.util.file_util.get_file_size_in_mb(file)

file may be the path to a file or a fileobject

The mp_cache Module

class cloudfusion.util.mp_cache.MPCache(expiration_time=60)

Bases: cloudfusion.util.cache.Cache

Multiprocessing cache that can synchronize entries over multiple processes.

Return an MPCache instance.

Parameters:expiration_time – Time in seconds until entries are expired.
get_size_of_cached_data()
get_size_of_dirty_data()

The xmlparser Module

Parses an XML file or string, populating a dictionary containing the keys with the desired values.

The keys of the dictionary are either the names of Tags contained in the XML data or again dictionaries. The values can either be dictionaries again or simple strings, but ultimately the leaf values will be strings.

class cloudfusion.util.xmlparser.DictXMLParser

Bases: object

_DictXMLParser__populate_dict(dom_tree, dict_tree)

Does the work for populate_dict_with_XML_leaf_textnodes()

Parameters:
  • dom_tree – An XML element
  • dict_tree – A dictionary of keys/values matching the first XML subtree in dom_tree
_DictXMLParser__populate_dict_with_collections(dom_tree, dict_tree)

Does the work for populate_dict_with_XML_collection_leaf_textnodes()

Parameters:
  • dom_tree – An XML element
  • dict_tree – A dictionary of keys/values matching the first XML subtree in dom_tree
getText(node)
get_elements_by_tag_name(dom_tree, tag_name)
Returns:elem a list of elements matching the tags with the name tag_name on the first level of dom_tree.
populate_dict_with_XML_collection_leaf_textnodes(xml_tree, dict_tree)

Puts the text of the XML tags of xml_tree specified by dict_tree into dict_tree .

The dict_tree is a dictionary describing a subtree of xml_data. The dictionarie’s keys correspond to the first XML tags with the same name on the same nesting level of the dictionary. Additionally, keys with the name surrounded by brackets are seen as a list of such XML tags. Therefore the keys in the dictionary {“root”: {“[element]”: “”}} match the root tag <root> as well as the subelements of root <element> and <element>. The dictionarie’s most deeply nested elements must be empty strings. This value is replaced by the text of the corresponding key’s XML tag. Given the previous dictionary and the following XML structure:

<root>
    <element>text1</element>
    <element>text2</element>
</root>

The resulting dictionary would be {“root”: {“[element]”: [“text1”, “text2”]}}

Parameters:
  • xml_tree – An XML string or an XML file
  • dict_tree – A dictionary of keys/values matching the first XML subtree in xml_tree
populate_dict_with_XML_leaf_textnodes(xml_data, dict_tree)

Puts the text of the XML tags of xml_data specified by dict_tree into dict_tree. The dict_tree is a dictionary describing a subtree of xml_data. The dictionarie’s keys correspond to the first XML tags with the same name on the same nesting level of the dictionary. Therefore the keys in the dictionary {“root”: {“element1”: “”, “element2”: “”}} match the root tag <root> as well as the subelements of root <element1> and <element2>. The dictionarie’s most deeply nested elements must be empty strings. This value is replaced by the text of the corresponding key’s XML tag. Given the previous dictionary and the following XML structure:

<root>
    <element1>text1</element1>
    <element2>text2</element2>
</root>

The resulting dictionary would be {“root”: {“element1”: “text1”, “element2”: “text2”}}

Parameters:
  • xml_data – An XML string or an XML file
  • dict_tree – A dictionary of keys/values matching the first XML subtree in xml_tree

The synchronize_proxy Module

class cloudfusion.util.synchronize_proxy.SynchronizeMethodWrapper(obj, func, name, rlock)

Wrapper object for a method to be called.

class cloudfusion.util.synchronize_proxy.SynchronizeProxy(core, private_methods_to_synchronize=[])

Bases: object

Proxy object that synchronizes access to a core object methods and attributes that don’t start with _.

The persistent_lru_cache Module

Created on Feb 5, 2013

class cloudfusion.util.persistent_lru_cache.PersistentLRUCache(directory, expiration_time=60, maxsize_in_MB=2000)

Bases: cloudfusion.util.lru_cache.LRUCache

A persistent LRU cache for restricting memory usage to a minimum and carrying the cache information over several sessions even after application crashes.

Return an LRUCache instance.

Parameters:
  • expiration_time – Time in seconds until entries are expired.
  • maxsize_in_MB – Approximate limit of the cache in MB.
  • directory – Directory to store persistent data, also serves as identifier for a persistent cache instance.
_close()
_get_file_content(filepath)
_get_persistent_size(filepath)
_get_size_of_entry(entry)
_write_to_file(filename, key, value)
delete(key)

Remove current entry associated with key from the LRU queue and delete its persistent representation.

get_keys()
get_size_of_dirty_data()
get_value(key)
peek(key)
peek_file(key)

Like peek, but memory efficient The temporary file needs to be closed if it is not used anymore :returns: temporary file object with the value

refresh(key, disk_value, modified)

Refreshes an entry with disk_value, if modified is bigger than the entry’s modified date.

set_modified(key, modified)
write(key, value)

The string Module

class cloudfusion.util.string.RegExData(regex, matchnr=1, type='int')
cloudfusion.util.string.__num_to_alpha(num)

From: http://stackoverflow.com/questions/10326118/encoding-a-numeric-string-into-a-shortened-alphanumeric-string-and-back-again

cloudfusion.util.string.get_id_key(dictionary)

Get the key for the id defined in dictionary from a list of synonyms. Cloud providers have different names for the id like amazon’s access_key_id or Dropbox’ consumer_key. :returns: key from dicitionary which is a synonym for an cloud account id or None otherwise

cloudfusion.util.string.get_secret_key(dictionary)

Get the key for the secret defined in dictionary from a list of synonyms. Cloud providers have different names for the secret like amazon’s secret_access_key or Dropbox’ consumer_secret. :returns: key from dicitionary which is a synonym for an cloud account secret or None otherwise

cloudfusion.util.string.get_uuid()

Get globally unique identifier

cloudfusion.util.string.indentLines(string, indent)

Indents a string’s lines by indent spaces. Does not indent empty strings.

cloudfusion.util.string.regSearchInt(needle, haystack, grp=1)
cloudfusion.util.string.regSearchString(needle, haystack, grp=1)
cloudfusion.util.string.to_unicode(text, encoding)

The mp_synchronize_proxy Module

class cloudfusion.util.mp_synchronize_proxy.MPSynchronizeProxy(core, private_methods_to_synchronize=[])

Bases: object

Proxy object that synchronizes access to a core object methods and attributes that don’t start with _ for multiple processes.

class cloudfusion.util.mp_synchronize_proxy.SynchronizeMethodWrapper(obj, func, name, rlock)

Wrapper object for a method to be called.

The cache Module

Created on 03.06.2011

class cloudfusion.util.cache.Cache(expiration_time)

Bases: object

Return a Cache instance with entries expiring after expiration_time seconds.

_get_size_of_entry(entry)
delete(key)
exists(key)
flush(key)
get_keys()
get_modified(key)
get_size_of_cached_data()
get_size_of_dirty_data()
get_value(key)
is_dirty(key)
is_expired(key)
refresh(key, disk_value, modified)

Refreshes an entry with disk_value, if modified is bigger than the entry’s modified date.

set_dirty(key, is_dirty)
set_modified(key, modified)
update(key)
write(key, value)
class cloudfusion.util.cache.Entry(value, dirty, modified=None, updated=None)

Bases: object

Cache entry.

The lru_cache Module

Created on Feb 5, 2013

class cloudfusion.util.lru_cache.LRUCache(expiration_time=60, maxsize_in_MB=200)

Bases: cloudfusion.util.cache.Cache

Least Recently Used cache. When adding an entry to the cache exceeds its size limit, the cache discards dirty entries starting with the least recently used. However, dirty entries or the last entry in the cache are not deleted.

Return an LRUCache instance.

Parameters:
  • expiration_time – Time in seconds until entries are expired.
  • maxsize_in_MB – Approximate limit of the cache in MB.
_get_listhead_entry()
_get_listtail_entry()
_move_used_entry_to_head(key)

Put existing entry associated with key in front of the LRU queue.

_resize()

Resize cache to maxsize.

_store_to_dict(linkedEntry)

This is only important for subclasses using shelve with writeback=False as a dictionary.

delete(key)

Remove current entry associated with key from the LRU queue.

flush(key)
get_dirty_lru_entries(num)

Get list of num dirty least recently used entries. returns: list of keys of the num least recently used entries

get_keys()
get_resize_intervall()
Returns:The minimal interval between consecutive calls to reduce cache size as a float in seconds. The default is one second.
get_size_of_cached_data()
get_size_of_dirty_data()
get_value(key)
peek(key)

Get value associated with key. This does not count as the entry being used.

refresh(key, disk_value, modified)

Refreshes an entry with disk_value, if modified is bigger than the entry’s modified date.

set_dirty(key, is_dirty)
set_resize_intervall(seconds)

Do not try to reduce the cache size again before seconds have passed. Resize operations are slow, especially when the cache has many elements. It can be set to 0, if you want to make sure that clean entries are removed as fast as possible. :param seconds: float stating how much time in seconds must pass before a cache size reduction is tried again

write(key, value)
class cloudfusion.util.lru_cache.LinkedEntry(value, dirty, modified=None, updated=None, key=None, next=None, prev=None)

Bases: cloudfusion.util.cache.Entry

Cache entry with next and previous links for chaining into a doubly linked list and a key for direct reference.