API

Discovery, filtering, grouping and slicing of model history files.

Developer: Cameron Cummins Contact: cameron.cummins@utexas.edu

class gents.hfcollection.HFCollection(hf_dir, num_processes=1, meta_map=None, hf_groups=None, step_map=None, hf_glob_pattern='*.nc*', dask_client=None, multistep_slice_map={})

A set of history files and their metadata.

Holds a {path: netCDFMeta | None} mapping whose metadata is read lazily by pull_metadata(), so path-glob filters can run before any file is opened. Every filter and transform returns a new instance, and a copy inherits the metadata already read.

Built for viable history files: files without a usable time coordinate are dropped and a group holding a single time step raises. Use find_files() / sort_hf_groups() directly to inspect a raw case tree.

check_pulled()

Pulls metadata if it has not been pulled already.

check_validity()

Drops files whose metadata is missing or not is_valid(), warning about each.

Returns:

The removed {path: metadata} entries.

Return type:

dict

copy(num_processes=None, meta_map=None, hf_groups=None, step_map=None, multistep_slice_map=None)

Returns a new collection derived from this one, with optional overrides.

Shares the original’s hf_dir. Every filter and transform returns through here, which is what keeps the API immutable. Arguments left None are inherited.

Parameters:

  • num_processes (int or None) – Worker process count for the copy.

  • meta_map (dict or None) – Metadata map to assign to the copy.

  • hf_groups (dict or None) – Group mapping to assign to the copy.

  • step_map (dict or None) – Timestep delta mapping to assign to the copy.

  • multistep_slice_map (dict or None) – Multistep slice indices to assign to the copy.

Return type:

HFCollection

exclude(glob_patterns)

Returns a new collection with files matching any pattern removed.

Patterns are fnmatch globs tested against absolute path strings.

Parameters:

glob_patterns (list[str] or str) – One or more glob patterns; a single string is also accepted.

Return type:

HFCollection

exclude_patterns(glob_patterns)

Deprecated since version Use: exclude() instead.

get_groups(check_fragmented=True)

Returns the collection’s {group ID: [paths]} mapping.

Groups are built by sort_hf_groups() on the first call and cached.

Parameters:

check_fragmented (bool) – Also merge spatially tiled groups via merge_fragmented_groups(), which requires metadata.

Return type:

dict[str, list[pathlib.Path]]

get_input_dir()

Returns the head directory this collection was initialised from.

Return type:

str

get_multistep_slices(hf_path)

Returns where to cut a multi-timestep history file that straddles a slice boundary, or None if it does not need cutting.

Parameters:

hf_path (pathlib.Path) – Path to the history file.

Returns:

{'<start>-<end>': (start_index, end_index)}, one entry per slice the file contributes to.

Return type:

dict or None

Raises:

KeyError – If the path is not in this collection.

get_timestep_delta(hf_path)

Returns the duration of one time step in the file’s group, pulling metadata first if necessary.

Parameters:

hf_path (pathlib.Path) – Path to the history file.

Return type:

datetime.timedelta

include(glob_patterns)

Returns a new collection holding only files matching at least one pattern.

Patterns are fnmatch globs tested against absolute path strings. An empty list matches nothing and so empties the collection.

Parameters:

glob_patterns (list[str] or str) – One or more glob patterns; a single string is also accepted.

Return type:

HFCollection

include_patterns(glob_patterns)

Deprecated since version Use: include() instead.

include_years(start_year, end_year, glob_patterns=['*'])

Returns a new collection holding only files within a year range.

A file’s year is the midpoint of its first time bound, or its first time value when it has no bounds. Metadata is pulled if necessary, so prefer include() when a path filter would do.

Parameters:

  • start_year (int) – First year in the range (inclusive).

  • end_year (int) – Last year in the range (inclusive).

  • glob_patterns (list[str]) – Restricts which files the year filter applies to.

Return type:

HFCollection

is_pulled()

Returns whether metadata has been loaded for every file in the collection.

Return type:

bool

pull_metadata(check_valid=True, raise_errors=False, show_progress=True)

Reads the header of every history file in the collection.

Runs get_meta_from_path() over a process pool when num_processes > 1 and serially otherwise, then computes each group’s timestep delta.

Parameters:

  • check_valid (bool) – Run check_validity() afterwards, dropping files with invalid or incomplete metadata.

  • raise_errors (bool) – Raise per-file failures instead of logging them and carrying on.

  • show_progress (bool) – If False, suppress the stdout progress bar.

Raises:

ValueError – If any group holds fewer than two time steps in total, regardless of raise_errors.

slice_groups(slice_size_years=10, start_year=0, pattern='*', time_alignment_method='midpoint')

Returns a new collection with its groups partitioned into year windows.

Each group is split into windows of slice_size_years, and every file assigned to the window its representative time falls in. Files straddling a boundary record per-slice cut points (see get_multistep_slices()). Sub-group keys gain a [sorting_pivot]<start>-<end> suffix, which TSCollection parses back out.

Parameters:

  • slice_size_years (int) – Maximum width of each window in years.

  • start_year (int or None) – Year to align windows to; None uses the collection’s own earliest year.

  • pattern (str) – fnmatch glob restricting which groups are sliced.

  • time_alignment_method (str) – How to pick a file’s representative time: 'midpoint' of its first time bound, 'direct_time' (ignoring bounds), 'start_bound' or 'end_bound'.

Return type:

HFCollection

Raises:

ValueError – If time_alignment_method is not one of those four.

sort_along_time()

Returns a new collection with its files ordered by their first time value.

Return type:

HFCollection

gents.hfcollection.calculate_year_slices(slice_size_years, min_year, max_year)

Computes non-overlapping year ranges covering a span.

Each range is at most slice_size_years wide, with the upper boundary rounded up to the next multiple of that width. A span no wider than one slice is returned unsliced.

Parameters:

  • slice_size_years (int) – Maximum width of each slice in years.

  • min_year (int) – First year in the range (inclusive).

  • max_year (int) – Last year in the range (inclusive).

Returns:

List of (start_year, end_year) tuples, one per slice.

Return type:

list[tuple[int, int]]

Raises:

ValueError – If max_year is less than min_year.

gents.hfcollection.check_groups_by_variables(sliced_groups)

Drops files whose variable set differs from the majority of their group.

Minority files are discarded with a warning and the survivors re-sorted by time; a group with no clear majority is dropped entirely.

Parameters:

sliced_groups (dict) – {group ID: [netCDFMeta]} to filter.

Returns:

The same mapping, holding only majority-consistent files.

Return type:

dict

gents.hfcollection.filter_by_variables(meta_datasets)

Splits history files by variable set, majority against the rest.

Files are fingerprinted by their sorted variable names. If no single fingerprint is most common, both returned values are None.

Parameters:

meta_datasets (list[gents.meta.netCDFMeta]) – Metadata objects to examine.

Returns:

(majority, others); others is None when every file shares the same variable set.

Return type:

tuple[list or None, list or None]

gents.hfcollection.find_files(head_path, pattern)

Recursively finds files whose names match an fnmatch pattern.

Parameters:

  • head_path (str or pathlib.Path) – Root directory to search.

  • pattern (str) – Wildcard pattern matched against file names (e.g. '*.nc').

Returns:

Sorted list of matching paths.

Return type:

list[pathlib.Path]

gents.hfcollection.get_group_timestep_delta(metas)

Computes the duration of one time step for a group of history files.

The duration is the gap between the two latest time values in the group, i.e. the resolution at the end of the record. Fragmented tiles share every time step, so their two latest values are identical and the delta is zero.

Only the two latest steps per file are compared as CFTime objects; each file’s pair is picked in linear time from its raw float times, which are monotonic with the decoded values. That avoids an object-dtype sort over every step in the group while still ordering files with differing units/calendar correctly.

Parameters:

metas (list[gents.meta.netCDFMeta]) – Metadata objects for the history files in one group.

Returns:

Duration of one time step.

Return type:

datetime.timedelta

Raises:

ValueError – If the group holds fewer than two time steps in total.

gents.hfcollection.get_year_boundary_num(year, units, calendar)

Returns the raw time value of midnight, January 1 of year under the given time reference.

Comparing raw time values against these boundaries reproduces year-based tests (start <= time.year <= end) without decoding every time step. A year the calendar cannot represent (year 0 in a standard calendar) steps forward to the nearest representable year, which selects the same set of times: no time value can fall inside the missing year.

Parameters:

  • year (int) – Calendar year of the boundary.

  • units (str) – CF time units the result is expressed in.

  • calendar (str) – CF calendar name.

Returns:

Boundary expressed as a raw time value.

Return type:

float

gents.hfcollection.get_year_bounds(hf_to_meta_map)

Returns the (min_year, max_year) covered by a set of history files.

A file’s year comes from the midpoint of each time bound, or from the time value itself when the file has no bounds.

Parameters:

hf_to_meta_map (dict) – {path: netCDFMeta} mapping to inspect.

Return type:

tuple[int, int]

gents.hfcollection.is_ds_within_years(ds_meta, min_year, max_year)

Returns whether a file’s representative time falls within a year range.

The representative year is the midpoint of the first time bound, or the first time value when the file has no bounds.

Parameters:

  • ds_meta (gents.meta.netCDFMeta) – Metadata object for the file to check.

  • min_year (int) – Lower bound of the range (inclusive).

  • max_year (int) – Upper bound of the range (inclusive).

Return type:

bool

gents.hfcollection.merge_fragmented_groups(hf_groups, hf_meta_map)

Merges spatially fragmented (tiled) history file groups into single groups.

A group is taken to be fragmented when its first path does not end in .nc (tiles look like *.nc.0001). Fragmented groups sharing the same non-time dimension bounds are merged under one wildcard key; other groups pass through unchanged.

Parameters:

  • hf_groups (dict) – {group pattern: [paths]} to merge.

  • hf_meta_map (dict) – {path: netCDFMeta}, used for dimension bounds.

Returns:

New group mapping with fragmented groups merged.

Return type:

dict

Raises:

KeyError – If a merged group’s label collides with an existing group.

gents.hfcollection.sort_hf_groups(hf_paths, delimiter='.', substring_index=2)

Groups history file paths by parent directory and shared filename prefix.

The prefix is the filename minus its last substring_index delimiter-separated tokens, so model.h0.0001-01.nc and model.h0.0001-02.nc both group under model.h0. A name with fewer tokens than that keeps what it has (gridfile.nc groups under gridfile*).

Ordering is part of the contract: keys come out ordered by parent directory (first appearance) then prefix, and paths keep their input order within a group. Downstream output order inherits it.

Parameters:

  • hf_paths (list[pathlib.Path]) – History file paths to group.

  • delimiter (str) – Token delimiter within the filename.

  • substring_index (int) – Number of trailing tokens to strip for the prefix.

Returns:

{'<parent_dir>/<prefix>*': [paths]}.

Return type:

dict[str, list[pathlib.Path]]

gents.hfcollection.sort_metas_by_time(metas)

Returns a new list of metadata objects sorted by their first time value.

Parameters:

metas (list[gents.meta.netCDFMeta]) – Unsorted metadata objects; not modified.

Return type:

list[gents.meta.netCDFMeta]

Per-file netCDF metadata: variable classification and cached header contents.

Developer: Cameron Cummins Contact: cameron.cummins@utexas.edu

gents.meta.get_attributes(dataset)

Extracts all attributes from a netCDF4 dataset or variable into a dictionary.

Parameters:

dataset (netCDF4.Dataset or netCDF4._netCDF4.Variable) – netCDF4 dataset or variable to read attributes from.

Returns:

Dictionary mapping attribute names to their values.

Return type:

dict

gents.meta.get_meta_from_path(path: str)

Opens a netCDF file and returns a netCDFMeta built from it.

Picklable factory, so metadata can be read inside ProcessPoolExecutor workers.

Parameters:

path (str) – Path to the netCDF history file.

Returns:

Metadata object populated from the file.

Return type:

netCDFMeta

Raises:

Exception – Re-raises construction errors with the path appended.

gents.meta.get_time_variables_names(ds)

Locates the time and time-bounds variable names in a netCDF dataset.

Matching is case-insensitive against time and, for bounds, time_bnds, time_bnd, time_bounds or time_bound.

Parameters:

ds (netCDF4.Dataset) – Open netCDF4 dataset to inspect.

Returns:

(time_name, time_bounds_name); either is None if not found.

Return type:

tuple[str or None, str or None]

gents.meta.is_var_secondary(variable, secondary_vars: list = ['time_bnds', 'time_bnd', 'time_bounds', 'time_bound'], secondary_dims: list = ['nbnd', 'chars', 'string_length', 'hist_interval'], max_num_dims: int = 1, primary_dims: list = ['time']) → bool

Classifies a netCDF variable as secondary or primary.

Primary variables are multi-dimensional, time-varying scientific fields; each gets its own time series file. Everything else (coordinates, bounds, metadata) is secondary and is copied into every output file of the group.

A variable is secondary if its name is in secondary_vars, if any of its dimensions is in secondary_dims, or if it has at most max_num_dims dimensions or none of primary_dims. All comparisons are case-insensitive, so MOM6-style Time / Time_Bounds are recognised.

Parameters:

  • variable (netCDF4._netCDF4.Variable) – netCDF4 variable object to classify.

  • secondary_vars (list) – Variable names that are unconditionally secondary.

  • secondary_dims (list) – Dimension names whose presence makes a variable secondary.

  • max_num_dims (int) – Dimension count at or below which a variable is secondary.

  • primary_dims (list) – Dimension names whose presence keeps a variable primary.

Returns:

True if the variable is secondary, False if primary.

Return type:

bool

class gents.meta.netCDFMeta(ds, path: str, decode_dates=True, load_time_bounds=True, load_variable_attrs=True, compute_dim_bounds=True)

Metadata read once from a single netCDF history file and cached in memory.

Picklable, so it can be built in a worker process and returned to the parent (see get_meta_from_path()).

decode_time_bounds_values(values)

Decodes raw time-bounds values into CFTime objects using the time-bounds variable’s reference (units and calendar).

Parameters:

values (numpy.ndarray or float) – Raw time-bounds value(s) expressed in the bounds units.

Returns:

Decoded CFTime object(s) matching the input’s shape.

Return type:

numpy.ndarray or cftime.datetime

Raises:

RuntimeError – If the file has no time-bounds variable, or it was not loaded (load_time_bounds=False).

decode_time_values(values)

Decodes raw time values into CFTime objects using this file’s time reference (units and calendar).

Parameters:

values (numpy.ndarray or float) – Raw time value(s) expressed in this file’s time units.

Returns:

Decoded CFTime object(s) matching the input’s shape.

Return type:

numpy.ndarray or cftime.datetime

get_attributes()

Returns the global attributes cached from the history file.

Return type:

dict

get_cftime_bounds()

Returns the time-bounds array as CFTime pairs, or None if the file has no time-bounds variable.

Return type:

numpy.ndarray or None

Raises:

RuntimeError – If constructed with load_time_bounds=False and the file actually has a time-bounds variable.

get_cftimes()

Returns the time values as CFTime objects, one per time step.

Return type:

numpy.ndarray

get_dim_bounds()

Returns {dimension: [min]} or {dimension: [min, max]} for every dimension that has a coordinate variable.

Used by merge_fragmented_groups() to match the spatial extent of tiled files.

Return type:

dict

Raises:

RuntimeError – If constructed with compute_dim_bounds=False.

get_float_time_bounds()

Returns the time-bounds array as raw float pairs, or None if the file has no time-bounds variable.

Return type:

numpy.ndarray or None

Raises:

RuntimeError – If constructed with load_time_bounds=False and the file actually has a time-bounds variable.

get_float_times()

Returns the raw float time values read from the time variable.

Return type:

numpy.ndarray

get_path()

Returns the path of the history file this object was built from.

Return type:

str

get_primary_variables()

Returns the names of the primary variables (see is_var_secondary()).

Return type:

list[str]

get_secondary_variables()

Returns the names of the secondary variables (see is_var_secondary()).

Return type:

list[str]

get_time_bounds_calendar()

Returns the calendar attribute of the time-bounds variable (falling back to the time variable’s at load time), or None if the file has none.

Return type:

str or None

get_time_bounds_units()

Returns the units attribute of the time-bounds variable (falling back to the time variable’s at load time), or None if the file has none.

Return type:

str or None

get_time_calendar()

Returns the calendar attribute of the time variable.

Return type:

str

get_time_units()

Returns the units attribute of the time variable.

Return type:

str

get_time_var_name()

Returns the name of the time variable in the history file.

Return type:

str

get_timebnds_var_name()

Returns the name of the time-bounds variable, or None if there is none.

Return type:

str or None

get_variable_attrs(variable)

Returns the attribute dictionary of the given variable.

Parameters:

variable (str) – Name of the variable to look up.

Return type:

dict

Raises:

RuntimeError – If constructed with load_variable_attrs=False.

get_variable_dims(variable)

Returns the dimension names of the given variable.

Parameters:

variable (str) – Name of the variable to look up.

Return type:

tuple[str]

get_variable_dtype(variable)

Returns the NumPy dtype of the given variable.

Parameters:

variable (str) – Name of the variable to look up.

Return type:

numpy.dtype

get_variable_shapes(variable)

Returns the shape of the given variable in this file.

Parameters:

variable (str) – Name of the variable to look up.

Return type:

tuple[int]

get_variables()

Returns the names of every variable in the history file.

Return type:

list[str]

is_valid()

Returns whether this history file is usable for time series generation.

A file is invalid if it has no usable time coordinate, holds no variables, or carries a gents_version attribute (marking it as GenTS output rather than raw model output).

Return type:

bool

Construction and execution of time series generation orders, and file writing.

Developer: Cameron Cummins Contact: cameron.cummins@utexas.edu

class gents.timeseries.TSCollection(hf_collection, output_dir, ts_orders=None, num_processes=None, dask_client=None)

The set of time series generation orders derived from an HFCollection.

An order is a dictionary describing one output file: its source history file paths, output path template, primary and secondary variables, timestamp string, and any generation arguments added by the modifier methods. Every modifier returns a new TSCollection.

add_args(path_glob='*', var_glob='*', level=None, alg=None, overwrite=None, chunk_target_bytes=None)

Sets generation arguments on orders that match both filters.

Arguments left None are not applied. The other apply_* methods are thin wrappers around this one.

Parameters:

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

  • level (int or None) – netCDF4 compression level (0-9).

  • alg (str or None) – netCDF4 compression algorithm, e.g. 'zlib'.

  • overwrite (bool or None) – Whether matching outputs are overwritten.

  • chunk_target_bytes (int or None) – Chunk size target for write_timeseries_file().

Return type:

TSCollection

add_attrs(attrs)

Adds global attributes to every output file of this collection.

Parameters:

attrs (dict) – Attributes to stamp into the output files.

Return type:

TSCollection

append_timestep_dirs(var_glob='*')

Inserts a frequency directory (hour_6, month_1, …) before the filename of each matching order, organising output by frequency.

Orders that do not match var_glob are dropped, not just left alone.

Parameters:

var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

apply_chunk_target_bytes(target_bytes, path_glob='*', var_glob='*')

Sets the chunk size target on matching orders (see add_args()).

Output written with a non-default target will not pass check_timeseries_conform(), which always checks against CHUNK_TARGET_BYTES.

Parameters:

  • target_bytes (int) – Target chunk size in bytes.

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

apply_compression(level, alg, path_glob, var_glob='*')

Applies compression settings to matching orders (see add_args()).

Parameters:

  • level (int) – netCDF4 compression level (0-9).

  • alg (str) – netCDF4 compression algorithm, e.g. 'zlib'.

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

apply_overwrite(path_glob, var_glob='*')

Enables overwriting of existing output for matching orders.

Parameters:

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

apply_path_swap(string_match, string_swap, path_glob='*', var_glob='*')

Replaces a substring in the output path template of matching orders.

Used to redirect output into a different directory structure, e.g. '/hist/' to '/proc/tseries/'.

Parameters:

  • string_match (str) – Substring to find in the output path template.

  • string_swap (str) – Replacement string.

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

copy(hf_collection=None, output_dir=None, ts_orders=None, num_processes=None)

Returns a new collection derived from this one, with optional overrides.

Every modifier returns through here, which is what keeps the API immutable. Arguments left None are inherited.

Parameters:

  • hf_collection (gents.hfcollection.HFCollection or None) – HFCollection to assign to the copy.

  • output_dir (str or None) – Output directory to assign to the copy.

  • ts_orders (list or None) – Order list to assign to the copy.

  • num_processes (int or None) – Worker process count for the copy.

Return type:

TSCollection

create_directories(exist_ok=True)

Creates the output directory tree for every order.

Parameters:

exist_ok (bool) – Do not raise when a directory already exists.

exclude(path_glob, var_glob='')

Returns a new collection with orders matching both filters removed.

An order is dropped if any of its source paths matches path_glob and its primary variable matches var_glob.

Parameters:

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

execute(optimize=True, optimize_batch_n=200, raise_errors=False, no_data=False, show_progress=True, memory_limit_bytes=4294967296)

Runs every order, writing the time series files.

Orders sharing a first source file and time slice are batched together so each group of history files is opened once rather than once per variable. Work runs over a process pool when num_processes > 1 and in-process otherwise; per-order failures are logged and the rest of the run continues.

Parameters:

  • optimize (bool) – Batch orders sharing source files into single worker calls, rather than submitting one call per order.

  • optimize_batch_n (int) – Maximum number of variables per batch.

  • raise_errors (bool) – Raise order failures instead of logging them.

  • no_data (bool) – Skip reading and writing primary variable data, producing the full structure with primaries reading back as their fill value (see write_timeseries_file()).

  • show_progress (bool) – If False, suppress the stdout progress bar.

  • memory_limit_bytes (float) – Cache ceiling for each MHFDataset opened, per worker.

Returns:

Paths to every generated time series file.

Return type:

list[str]

get_hf_collection()

Returns the HFCollection this collection was derived from.

Return type:

gents.hfcollection.HFCollection

get_output_dir()

Returns the root directory generated time series are written to.

Return type:

str

include(path_glob, var_glob='*')

Returns a new collection holding only orders that match both filters.

An order is kept if any of its source paths matches path_glob and its primary variable matches var_glob.

Parameters:

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

remove_overwrite(path_glob, var_glob='*')

Disables overwriting of existing output for matching orders.

Parameters:

  • path_glob (str) – fnmatch glob applied to source history file paths.

  • var_glob (str) – fnmatch glob applied to primary variable names.

Return type:

TSCollection

update_ts_orders(strfrmt_kwargs={}, time_alignment_method='midpoint')

Rebuilds the order list and returns a new TSCollection.

One order is built per primary variable per group. The output path template comes from the group key with the input head directory and any [sorting_pivot] suffix stripped; hist-to-tseries style renaming is apply_path_swap()’s job, applied afterwards.

Parameters:

  • strfrmt_kwargs (dict) – Timestamp format overrides forwarded to get_timestamp_format(), e.g. {'monthly_format': '%Y%m%d'}.

  • time_alignment_method (str) – How to pick the representative time for the filename timestamp: 'midpoint' of the time bound, 'direct_time' (ignoring bounds), 'start_bound' or 'end_bound'.

Return type:

TSCollection

Raises:

ValueError – If time_alignment_method is not one of those four.

gents.timeseries.check_timeseries_conform(ts_path: str)

Checks whether a time series file meets the GenTS chunking convention.

A conforming file stores time contiguously, and every other variable either contiguously or in time chunks of at least CHUNK_TARGET_BYTES. Always checked against that constant, so a file written with a custom chunk_target_bytes will not conform.

Parameters:

ts_path (str) – Path to the time series file to inspect.

Return type:

bool

gents.timeseries.check_timeseries_integrity(ts_path: str)

Checks whether a time series file was written completely by GenTS.

The gents_version attribute is stamped last, so its presence means the write finished.

Parameters:

ts_path (str) – Path to the time series file to inspect.

Returns:

False if the stamp is absent or the file cannot be opened.

Return type:

bool

gents.timeseries.compute_chunksizes(var_shape, itemsize, target_bytes=4194304)

Chooses netCDF chunk sizes for a variable shaped (time, ...).

A variable smaller than target_bytes is stored contiguously; a larger one is chunked along the time axis, as many steps per chunk as fit in the target, with the remaining dimensions kept whole.

Parameters:

  • var_shape (list[int]) – Full variable shape, time axis first.

  • itemsize (int) – Size in bytes of one array element.

  • target_bytes (int) – Chunk size target, 4 MiB by default.

Returns:

Chunk sizes, one per dimension of var_shape.

Return type:

list[int]

gents.timeseries.generate_time_series(hf_paths, ts_path_template, secondary_vars, ts_args, no_data=False, memory_limit_bytes=4294967296)

Generates every time series file for one group of history files.

Opens the group once as an MHFDataset, reads the secondary variables, then writes one file per primary variable in ts_args. This is the unit of work submitted to the process pool by TSCollection.execute(), so its arguments must stay picklable.

Parameters:

  • hf_paths (list[str or pathlib.Path]) – Paths to the history files forming the group.

  • ts_path_template (str) – Output path prefix, without variable or timestamp.

  • secondary_vars (list[str]) – Secondary variables to embed in every output file.

  • ts_args (dict) – {primary variable: kwargs} for write_timeseries_file(); each must carry a 'ts_string' key holding the timestamp suffix.

  • no_data (bool) – Skip reading and writing primary variable data.

  • memory_limit_bytes (float) – Cache ceiling for the MHFDataset.

Returns:

Paths to the generated time series files.

Return type:

list[str]

gents.timeseries.get_timestamp_format(dt, subhour_format='%Y%m%d%H%M%S', hourly_format='%Y%m%d%H', daily_format='%Y%m%d', monthly_format='%Y%m', yearly_format='%Y')

Returns the strftime format to timestamp output files of a given frequency.

Parameters:

  • dt (datetime.timedelta) – Duration of a single model time step.

  • subhour_format (str) – Format for sub-minute steps.

  • hourly_format (str) – Format for steps under 24 hours.

  • daily_format (str) – Format for steps under 28 days.

  • monthly_format (str) – Format for steps under 12 months.

  • yearly_format (str) – Format for anything longer.

Return type:

str

gents.timeseries.get_timestep_label(dt)

Returns the frequency label for a time-step duration: 'hour_N', 'day_N', 'month_N', 'year_N', or 'unsorted' if unknown.

Used by TSCollection.append_timestep_dirs() as a directory name.

Parameters:

dt (datetime.timedelta or None) – Duration of a single model time step, or None if unknown.

Return type:

str

gents.timeseries.write_timeseries_file(agg_hf_ds, ts_out_path, primary_var, secondary_vars_data, overwrite=False, complevel=0, compression=None, ts_start_index=None, ts_end_index=None, append_attrs=None, no_data=False, chunk_target_bytes=4194304)

Writes one time series file, holding one primary variable and every secondary.

An existing output file is deleted and rewritten when overwrite is set; otherwise it is kept and skipped if it passes check_timeseries_integrity(), and deleted as corrupt if it does not. The completed file is stamped with a gents_version attribute last.

Every variable is created with the source’s _FillValue (if any) and any slice that is entirely that fill value is left unwritten, which is what keeps time series built from missing-value clones as small as their inputs. For ordinary data it is a no-op.

Parameters:

  • agg_hf_ds (gents.mhfdataset.MHFDataset) – Open MHFDataset for the group.

  • ts_out_path (str) – Full output path for the time series file.

  • primary_var (str) – Primary variable to write, or 'auxiliary' to write only the secondary variables.

  • secondary_vars_data (dict) – Pre-loaded {var_name: array} secondary data.

  • overwrite (bool) – Overwrite an existing output file rather than skip it.

  • complevel (int) – netCDF4 compression level (0-9).

  • compression (str or None) – netCDF4 compression algorithm, e.g. 'zlib'.

  • ts_start_index (int or None) – First time index to read from the group; None starts at the beginning.

  • ts_end_index (int or None) – Time index to stop reading at; None reads to the end.

  • append_attrs (dict or None) – Extra global attributes to stamp into the output.

  • no_data (bool) – Create the primary variable but neither read nor write its data, leaving it to read back as its fill value. The output stays structurally valid and self-describing; used for conformity runs over missing-value clones.

  • chunk_target_bytes (int) – Chunk size target passed to compute_chunksizes(). A non-default value will fail check_timeseries_conform().

Returns:

Path to the written (or skipped) output file.

Return type:

str

Virtual dataset presenting a group of history files as one aggregated whole.

Developer: Cameron Cummins Contact: cameron.cummins@utexas.edu

class gents.mhfdataset.MHFDataset(hf_paths, preload_var_list=None, memory_limit_bytes=inf, load_secondaries=True, preload_primaries=True)

Aggregating dataset interface over a group of related history files.

Presents files covering successive time steps and/or different spatial tiles as one virtual dataset. Files are opened one at a time – on open() (or __enter__) to read metadata and fill the cache, and again on demand for data that did not fit – so the number of open handles never scales with the size of the group.

Data is served from an in-memory cache keyed by variable. open() fills it with the secondary variables and as many of preload_var_list as fit under memory_limit_bytes; anything else is read (and cached, if it fits) on first use. A variable’s cache is dropped when reads move on to the next variable.

close()

Drops all cached variable data, leaving the instance ready to be reopened.

get_global_attrs()

Returns the global attributes of every file in the group, merged with later files winning on conflicting keys.

Return type:

dict

get_time_vals()

Returns the sorted, unique float time values across the group.

Served from the copy computed at open(); callers must not mutate the returned array.

Return type:

numpy.ndarray

get_var_attrs(var_name)

Returns a variable’s attributes, taken from the first file in the group.

Parameters:

var_name (str) – Name of the variable to inspect.

Return type:

dict

get_var_data_shape(var_name)

Returns a variable’s aggregated shape across the whole group.

Accounts for the total number of time steps and, for fragmented groups, the combined spatial extent. Coordinate variables get a single-element shape.

Parameters:

var_name (str) – Name of the variable to inspect.

Return type:

list[int]

get_var_dimensions(var_name)

Returns a variable’s dimension names, taken from the first file in the group.

Parameters:

var_name (str) – Name of the variable to inspect.

Return type:

tuple[str]

get_var_dsize(var_name)

Returns the size in bytes of a variable aggregated across the group.

Parameters:

var_name (str) – Name of the variable to size.

Return type:

int

get_var_dtype(var_name)

Returns a variable’s NumPy dtype, taken from the first file in the group.

Parameters:

var_name (str) – Name of the variable to inspect.

Return type:

numpy.dtype

get_var_vals(var_name, time_index_start=0, time_index_end=None)

Reads a variable’s data across the group for a slice of the time axis.

Non-fragmented groups are read in maximal runs of consecutive time steps that fall in the same file, one slice read per run. Fragmented groups are assembled step by step, each tile placed into a pre-allocated array by matching its coordinates against the group’s combined coordinate map.

Parameters:

  • var_name (str) – Name of the variable to read.

  • time_index_start (int) – First time step to include (inclusive).

  • time_index_end (int or None) – Last time step to include (exclusive); None reads to the end.

Returns:

Array of the variable’s data over the requested slice.

Return type:

numpy.ndarray

is_fragmented()

Returns whether the group is spatially fragmented, i.e. whether its first time value is covered by more than one file.

Return type:

bool

is_time_consistent()

Returns whether every time step is covered by the same number of files, i.e. no spatial tile is missing from any step.

Return type:

bool

open()

Reads every file in the group once, building the time mapping and cache.

__time_mapping maps each unique float time value to the (file_index, sub_time_index) pairs holding it, where sub_time_index is that value’s position within its own file’s time array – precomputed so get_var_vals() never rescans a time array.

Secondary variables are cached per file rather than read once from the first file, since time, bounds and per-tile coordinates differ between files; they are small enough for that to be cheap. Preloaded primaries ride along in the same pass so no file has to be reopened for them.

Raises:

Exception – If the spatial fragmentation is not consistent over time.

gents.mhfdataset.extend_coords(ds, dim_coords={})

Folds one dataset’s coordinates into a combined coordinate map.

Called once per file to build the group’s full extent: coordinate values are merged and de-duplicated across files, and a dimension with no coordinate variable gets a 0-indexed integer range instead.

Parameters:

  • ds (gents.datastore.GenTSDataStore or netCDF4.Dataset) – Open dataset to merge in.

  • dim_coords (dict) – Coordinate map accumulated from earlier files.

Returns:

The updated {dimension: coordinate array} map.

Return type:

dict

Logging setup, progress reporting, versioning, and collection summaries.

Developer: Cameron Cummins Contact: cameron.cummins@utexas.edu

class gents.utils.ProgressBar(total, length=40, label='', quiet=False)

Terminal progress bar drawn by overwriting a single stdout line in place.

Drawing is skipped entirely when stdout is not a terminal. Overwriting in place only works on one, so in a log file, a CI job or a pipe every redraw would land as another line and bury the output worth reading.

step()

Advances the bar by one iteration and redraws it, writing a trailing newline once the counter reaches total.

gents.utils.enable_logging(verbose=False, output_path=None)

Configures the gents logger to emit to stdout, and optionally to a file.

Parameters:

  • verbose (bool) – Log at LOG_LEVEL_IO_WARNING (5), which adds per-file I/O traces, instead of DEBUG. There is no quieter setting.

  • output_path (str or None) – File to write log output to in addition to stdout.

gents.utils.get_time_stamp()

Returns the current date and time as a 'YYYY-MM-DD HH:MM' string.

Return type:

str

gents.utils.get_version()

Returns the version of the installed gents package.

Return type:

str

gents.utils.is_terminal(stream)

Returns whether a stream is an interactive terminal.

Streams that stand in for stdout do not all implement isatty, so a stream that cannot answer is treated as not a terminal.

Parameters:

stream (io.IOBase) – Stream to test, usually sys.stdout.

Return type:

bool

gents.utils.log_hfcollection_info(hfc, show_progress=True)

Logs summary statistics for an HFCollection at INFO level.

Reports the input directory, file and group counts, total mapped data volume, the largest groups by variable and file count, and the largest single-timestep variable. Sizes are approximations (per-file variable sizes times file counts), not exact totals. Pulls metadata if it has not been pulled already.

Parameters:
gents.utils.log_tscollection_info(tsc, show_progress=True)

Logs summary statistics for a TSCollection at INFO level.

Reports the output directory, the number of time series files to generate, and the largest of them (source variable, shape, dimensions, projected size). Auxiliary-only orders are skipped and sizes are estimates, not exact totals.

Parameters:

run_gents – command line driver for the history file to time series pipeline.

gents.cli.check_config(config_dict)

Asserts that a model YAML config has the required top-level keys.

Parameters:

config_dict (dict) – Parsed contents of a gents/configs/*.yaml file.

Raises:

AssertionError – If any required key is missing.

gents.cli.main()

Entry point for run_gents.

Loads the YAML config for --model from gents/configs/, builds an HFCollection and TSCollection from it (with command line flags replacing the config’s filters and slicing, or extending them under --append), and executes the result unless --dryrun was given.

Raises:

ValueError – If --model names an unknown model, or --compression is given without --level.

gents.cli.parse_arguments()

Parses run_gents command line arguments.

Run run_gents --help for the full list; each flag’s help text below is its documentation.

Returns:

Namespace populated with the parsed argument values.

Return type:

argparse.Namespace