GenTS documentation

Generate Time Series (GenTS) is an open-source Python library and tool for converting output from earth system models and global climate models in the “history file” format to the “time series” format. GenTS utilizes a simplified Python interface to make this common post-processing task as easy as possible and leverages parallelism for optimal performance.

GenTS consolidates the conversion of history files to time series files into four steps:

1. Detect and read the metadata for all history files into a HFCollection

2. Apply filters to include/exclude certain history files and then group them by subdirectory (such as model component) and file name pattern (such as an output stream).

3. Derive a TSCollection from the HFCollection and apply configurations/filters to obtain the desired time series files

4. Generate an embarrassingly parallel workload to write the time series files

These steps can be executed via a command-line interface (CLI) or via a Python script using the API. Model specific configurations for CESM3, CESM2, and E3SM are included in the CLI.

The GenTS framework is “model agnostic” to avoid fragile, “hard-coded” implementations that tend to break between model updates. This also means that GenTS works for multiple model configurations. Python scripts that use GenTS functions are provided for each model under gents/configs. These scripts are then called by the CLI.

• Installation
  • PyPI
  • Container
  • Source
• User Guide
  • Using the Command Line Interface (CLI)
  • Using the Python Package
  • Creating the HFCollection
  • Creating the TSCollection
  • Custom Dask Workflows with TSCollection
• Developer Guide
  • Core Design Choices
  • Unit Testing
  • Docker (recommended)
  • Local environment
  • Benchmarking
• API
  • HFCollection
    • HFCollection.check_pulled()
    • HFCollection.check_validity()
    • HFCollection.copy()
    • HFCollection.exclude()
    • HFCollection.exclude_patterns()
    • HFCollection.get_groups()
    • HFCollection.get_input_dir()
    • HFCollection.get_multistep_slices()
    • HFCollection.get_timestep_delta()
    • HFCollection.include()
    • HFCollection.include_patterns()
    • HFCollection.include_years()
    • HFCollection.is_pulled()
    • HFCollection.pull_metadata()
    • HFCollection.slice_groups()
    • HFCollection.sort_along_time()
  • calculate_year_slices()
  • check_config()
  • check_groups_by_variables()
  • filter_by_variables()
  • find_all_indices()
  • find_files()
  • generate_output_template()
  • get_default_config()
  • get_year_bounds()
  • is_ds_within_years()
  • merge_fragmented_groups()
  • sort_hf_groups()
  • sort_metas_by_time()
  • get_attributes()
  • get_meta_from_path()
  • get_time_variables_names()
  • is_var_secondary()
  • netCDFMeta
    • netCDFMeta.get_attributes()
    • netCDFMeta.get_cftime_bounds()
    • netCDFMeta.get_cftimes()
    • netCDFMeta.get_dim_bounds()
    • netCDFMeta.get_float_time_bounds()
    • netCDFMeta.get_float_times()
    • netCDFMeta.get_path()
    • netCDFMeta.get_primary_variables()
    • netCDFMeta.get_secondary_variables()
    • netCDFMeta.get_variable_dims()
    • netCDFMeta.get_variable_dtype()
    • netCDFMeta.get_variable_shapes()
    • netCDFMeta.get_variables()
    • netCDFMeta.is_valid()
  • TSCollection
    • TSCollection.add_args()
    • TSCollection.add_attrs()
    • TSCollection.append_timestep_dirs()
    • TSCollection.apply_compression()
    • TSCollection.apply_overwrite()
    • TSCollection.apply_path_swap()
    • TSCollection.copy()
    • TSCollection.create_directories()
    • TSCollection.exclude()
    • TSCollection.execute()
    • TSCollection.get_hf_collection()
    • TSCollection.get_output_dir()
    • TSCollection.include()
    • TSCollection.remove_overwrite()
    • TSCollection.update_ts_orders()
  • check_timeseries_conform()
  • check_timeseries_integrity()
  • generate_time_series()
  • get_timestamp_format()
  • write_timeseries_file()
  • MHFDataset
    • MHFDataset.close()
    • MHFDataset.get_global_attrs()
    • MHFDataset.get_time_vals()
    • MHFDataset.get_var_attrs()
    • MHFDataset.get_var_data_shape()
    • MHFDataset.get_var_dimensions()
    • MHFDataset.get_var_dtype()
    • MHFDataset.get_var_vals()
    • MHFDataset.is_fragmented()
    • MHFDataset.is_time_consistent()
    • MHFDataset.open()
  • get_concat_coords()
  • ProgressBar
    • ProgressBar.step()
  • enable_logging()
  • get_time_stamp()
  • get_version()
  • log_hfcollection_info()
  • log_tscollection_info()
  • main()
  • parse_arguments()