Core API

fetchez.core

This module is the core of the Fetchez library. It handles the initialization of fetchers, connection pooling, threading, and the base FetchModule class.

copyright:

3. 2010-2026 Regents of the University of Colorado

license:

MIT, see LICENSE for more details.

fetchez.core.fetches_callback(r)

Default callback for fetches processes. r: [url, local-fn, data-type, fetch-status-or-error-code]

Parameters:

r (List[Any])

fetchez.core.urlencode_(opts)

Encode opts for use in a URL.

Parameters:

opts (Dict)

Return type:

str

fetchez.core.urlencode(opts, doseq=True)

Encode opts for use in a URL.

Parameters:

• opts (Dict) – Dictionary of query parameters.

• doseq (bool, default: True) – If True, lists in values are encoded as separate parameters (e.g., {‘a’: [1, 2]} -> ‘a=1&a=2’).

Return type:

str

fetchez.core.xml2py(node)

Parse an xml file into a python dictionary.

Return type:

Optional[Dict]

fetchez.core.get_userpass(authenticator_url)

Retrieve username and password from netrc for a given URL.

Parameters:

authenticator_url (str)

Return type:

Tuple[Optional[str], Optional[str]]

fetchez.core.get_credentials(url, authenticator_url='https://urs.earthdata.nasa.gov')

Get user credentials from .netrc or prompt for input. Used for EarthData, etc.

Parameters:

• url (str)

• authenticator_url (str, default: 'https://urs.earthdata.nasa.gov')

Return type:

Optional[str]

class fetchez.core.iso_xml(url=None, xml=None, timeout=20, read_timeout=60)

Bases: object

Helper class for parsing ISO 19115 XML Metadata.

__init__(url=None, xml=None, timeout=20, read_timeout=60)

title()

Extract Title.

abstract()

Extract Abstract.

date()

Extract Date.

linkages()

Extract first valid download URL (specifically looking for Zips/Data).

polygon(geom=True)

Extract Bounding Box and return GeoJSON Polygon.

class fetchez.core.HttpFile(url, session=None, callback=None)

Bases: IOBase

A file-like object backed by an HTTP URL.

Translates read() calls into HTTP Range requests to fetch only needed bytes.

__init__(url, session=None, callback=None)

seek(offset, whence=0)

Change the stream position to the given byte offset.

offset

The stream position, relative to ‘whence’.

whence

The relative position to seek from.

The offset is interpreted relative to the position indicated by whence. Values for whence are:

• os.SEEK_SET or 0 – start of stream (the default); offset should be zero or positive

• os.SEEK_CUR or 1 – current stream position; offset may be negative

• os.SEEK_END or 2 – end of stream; offset is usually negative

Return the new absolute position.

tell()

Return current stream position.

read(size=-1)

class fetchez.core.Fetch(url, callback=<function fetches_callback>, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'}, verify=True, allow_redirects=True)

Bases: object

Fetch class to fetch ftp/http data files

Parameters:

• url (str)

• headers (Dict, default: {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'})

• verify (bool, default: True)

• allow_redirects (bool, default: True)

__init__(url, callback=<function fetches_callback>, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'}, verify=True, allow_redirects=True)

Parameters:

• url (str)

• headers (Dict, default: {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'})

• verify (bool, default: True)

• allow_redirects (bool, default: True)

fetch_req(method='GET', params=None, data=None, json=None, tries=5, timeout=30, read_timeout=120)

Fetch src_url and return the requests object (iterative retry).

Parameters:

• method (str, default: 'GET')

• params (Optional[Dict], default: None)

• data (Optional[Any], default: None)

• json (Optional[Dict], default: None)

• tries (int, default: 5)

• timeout (Optional[float], default: 30)

• read_timeout (Optional[float], default: 120)

Return type:

Optional[Response]

fetch_html(timeout=2)

Fetch src_url and return it as an HTML object.

fetch_xml(timeout=2, read_timeout=10)

Fetch src_url and return it as an XML object.

fetch_file(dst_fn, method='GET', params=None, datatype=None, overwrite=False, timeout=30, read_timeout=120, tries=5, check_size=True, verbose=True)

Fetch src_url and save to dst_fn with resume support.

Parameters:

dst_fn (str)

Return type:

int

fetch_ftp_file(dst_fn, params=None, datatype=None, overwrite=False)

Fetch an ftp file via ftplib with a progress bar.

fetchez.core.run_fetchez(modules, threads=3, global_hooks=None)

Run Fetchez in parallel with hooks.

• mod.hooks: Run ONLY on entries belonging to ‘mod’.

• global_hooks: Run on ALL entries combined.

Parameters:

• modules (List[FetchModule])

• threads (int, default: 3)

class fetchez.core.FetchModule(src_region=None, callback=<function fetches_callback>, hook=None, outdir=None, name='fetches', min_year=None, max_year=None, weight=1.0, uncertainty=0.0, params={}, **kwargs)

Bases: object

Base class for all fetch modules.

__init__(src_region=None, callback=<function fetches_callback>, hook=None, outdir=None, name='fetches', min_year=None, max_year=None, weight=1.0, uncertainty=0.0, params={}, **kwargs)

property hooks

Combine internal and external hooks in the correct execution order.

add_hook(hook_obj)

Add a hook instance at runtime.

run()

set run in a sub-module to populate results with urls

fetch_entry(entry, check_size=True, retries=5, verbose=True)

fill_results(entry)

fill self.results with the fetch module entry

add_entry_to_results(url, dst_fn, data_type, **kwargs)

Add fetch entries to results. any keyword/args can be added to results, but we need url, dst_fn and data_type.

class fetchez.core.HttpDataset(url=None, **kwargs)

Bases: FetchModule

Fetch an http file directly.

__init__(url=None, **kwargs)

run()

set run in a sub-module to populate results with urls

class fetchez.core.Scratch(url, path, datatype, **kwargs)

Bases: FetchModule

Scratch module that just fills the results with it’s own arguments.

__init__(url, path, datatype, **kwargs)

run()

set run in a sub-module to populate results with urls