Modules Base

fetchez.modules.base

This holds the FetchModule super class

copyright:

3. 2010-2026 Regents of the University of Colorado

license:

MIT, see LICENSE for more details.

class fetchez.modules.base.FetchModule(src_region=None, hook=None, outdir=None, min_year=None, max_year=None, weight=1.0, uncertainty=0.0, params=None, use_cache=True, **kwargs)

Bases: object

Base class for all Fetchez data modules.

name = 'base_module'

meta_category = 'Generic'

meta_desc = 'Base module class.'

meta_agency = 'Unknown'

meta_tags: List[Any] = []

meta_aliases: List[Any] = []

meta_urls: Dict[Any, Any] = {}

__init__(src_region=None, hook=None, outdir=None, min_year=None, max_year=None, weight=1.0, uncertainty=0.0, params=None, use_cache=True, **kwargs)

property hooks

Combine internal and external hooks in the correct execution order.

add_hook(hook_obj)

Add a hook instance at runtime.

run()

Override this method in a subclass to populate self.results.

fetch_entry(entry, check_size=True, retries=5, verbose=True)

Standardized method for fetching a single result entry.

add_entry_to_results(url, dst_fn, data_type, **kwargs)

Add fetch entries to results.

At minimum, url, dst_fn and data_type are required. Any additional keyword arguments will be added to the entry dictionary.

class fetchez.modules.base.HttpDataset(url=None, **kwargs)

Bases: FetchModule

Fetch an HTTP/HTTPS file directly from a URL.

name = 'url_fetcher'

meta_category = 'Generic'

meta_desc = 'Fetch a file directly from a URL.'

meta_resolution = 'N/A'

meta_license = 'N/A'

__init__(url=None, **kwargs)

run()

Override this method in a subclass to populate self.results.

class fetchez.modules.base.Scratch(url=None, path=None, datatype=None, **kwargs)

Bases: FetchModule

Scratch module that populates results directly from arguments.

name = 'scratch'

meta_category = 'Reference'

meta_desc = 'Testing module that injects direct arguments into the pipeline.'

meta_resolution = 'N/A'

meta_license = 'N/A'

__init__(url=None, path=None, datatype=None, **kwargs)

run()

Override this method in a subclass to populate self.results.