IODA
|
Namespaces | |
anonymous_namespace{AtlasDistribution.cc} | |
anonymous_namespace{DistributionUtils.cc} | |
anonymous_namespace{ObsSpace.cc} | |
anonymous_namespace{RoundRobin.cc} | |
chunking | |
A few chunking strategies for Variables. | |
detail | |
Engines | |
The backends that implement the ioda-engines functionality. | |
FillValuePolicies | |
Holds the different default fill values used in ioda files produced by different backends. | |
ioda | |
ObsStore | |
Selections | |
test | |
Types | |
Defines the type system used for manipulating IODA objects. | |
Classes | |
struct | FileFormatParameterTraitsHelper |
Helps with the conversion of FileFormat values to/from strings. More... | |
struct | ObsDtypeParameterTraitsHelper |
Helps with the conversion of ObsDtype values to/from strings. More... | |
class | Accumulator |
Calculates the sum of a location-dependent quantity of type T over locations held on all PEs, each taken into account only once even if it's held on multiple PEs. More... | |
class | Accumulator< std::vector< T > > |
Calculates the sums of multiple location-dependent quantities of type T over locations held on all PEs, each taken into account only once even if it's held on multiple PEs. More... | |
class | AtlasDistribution |
Distribution assigning each record to the process owning the Atlas mesh partition containing the location of the first observation in that record. More... | |
class | Distribution |
class for distributing obs across multiple process elements More... | |
class | DistributionFactory |
Distribution factory. More... | |
class | DistributionMaker |
A class able to instantiate objects of type T, which should be a subclass of Distribution. More... | |
class | GeneralDistributionAccumulator |
Implementation of the Accumulator interface suitable for any (possibly overlapping) distribution, but potentially less efficient than specialized implementations. More... | |
class | GeneralDistributionAccumulator< std::vector< T > > |
class | Halo |
Halo distribution. More... | |
class | InefficientDistribution |
Inefficient distribution. More... | |
class | InefficientDistributionAccumulator |
Implementation of the Accumulator interface suitable for the InefficientDistribution. More... | |
class | InefficientDistributionAccumulator< std::vector< T > > |
class | NonoverlappingDistribution |
Implements some methods of Distribution in a manner suitable for distributions storing each observation on one and only one process. More... | |
class | NonoverlappingDistributionAccumulator |
Implementation of the Accumulator interface suitable for the NonoverlappingDistribution. More... | |
class | NonoverlappingDistributionAccumulator< std::vector< T > > |
class | PairOfDistributions |
Represents a concatenation of locations and records from two distributions. More... | |
class | PairOfDistributionsAccumulator |
Implementation of the Accumulator interface suitable for the PairOfDistributions. More... | |
class | PairOfDistributionsAccumulator< std::vector< T > > |
class | ReplicaOfGeneralDistribution |
class | ReplicaOfNonoverlappingDistribution |
Distribution assigning each record to a process if and only if a non-overlapping master distribution has done the same. More... | |
class | RoundRobin |
Round robin distribution. More... | |
class | Attribute |
This class represents attributes, which may be attached to both Variables and Groups. More... | |
class | Attribute_Creator |
Flywheel creation of ioda::Attribute. More... | |
class | Attribute_Creator_Store |
Flywheel creation of ioda::Attribute objects.This is needed because you might want to make the same Attribute in multiple places. More... | |
class | Has_Attributes |
This class exists inside of ioda::Group or ioda::Variable and provides the interface to manipulating Attributes. More... | |
class | ObjectSelection |
Allows you to select objects for a copy operation. More... | |
struct | ScaleMapping |
Settings for how to remap dimension scales. More... | |
class | Exception |
The ioda exception class. More... | |
class | Group |
Groups are a new implementation of ObsSpaces. More... | |
struct | Dimensions |
Describes the dimensions of an Attribute or Variable. More... | |
struct | ScaleSizes |
struct | NewDimensionScale_Base |
Used to specify a new dimension scale when making an ObsSpace. More... | |
struct | NewDimensionScale_Object |
Used to specify a new dimension scale when making an ObsSpace. Templated version of NewDimensionScale_Base. More... | |
struct | ComplementaryVariableCreationParameters |
class | Options |
Quick and easy key-value container that stringifies all values. More... | |
class | ObsGroup |
An ObsGroup is a specialization of a ioda::Group. It provides convenience functions and guarantees that the ioda data are well-formed. More... | |
struct | Marshalled_Data |
Structure used to pass data between the frontend and the backend engine. More... | |
class | Type |
Represents the "type" (i.e. integer, string, float) of a piece of data. More... | |
struct | VariableCreationParameters |
Used to specify Variable creation-time properties. More... | |
struct | NewVariable_Base |
Used to specify a new variable with the collective createWithScales function. More... | |
class | Has_Variables |
This class exists inside of ioda::Group and provides the interface to manipulating Variables. More... | |
class | Selection |
A Selection represents the bounds of the data, in ioda or in userspace, that you are reading or writing. More... | |
class | Variable |
Variables store data! More... | |
struct | Named_Variable |
A named pair of (variable_name, ioda::Variable). More... | |
class | ObsFrame |
class | ObsFrameRead |
Implementation of ObsFrameRead class. More... | |
class | ObsFrameWrite |
Implementation of ObsFrameWrite class. More... | |
class | ObsIo |
class | ObsIoFactory |
class | ObsIoMaker |
class | ObsIoFileCreate |
Implementation of ObsIo creating a file. More... | |
class | ObsIoFileRead |
Implementation of ObsIo reading data from a file. More... | |
class | ObsIoGenerateList |
Implementation of ObsIo generating observations at locations specified in the input YAML file (parsed earlier into the ObsGenerateListParameters object passed to the constructor). More... | |
class | ObsIoGenerateRandom |
Implementation of ObsIo generating observations at random locations. More... | |
class | ObsGroupingParameters |
Options controlling the manner in which observations are grouped into records. More... | |
class | ObsIoParametersBase |
Base of classes storing the configuration parameters of ObsIo subclasses. More... | |
struct | IodaTrait |
class | TimeIodaIO |
class | ObsDataVector |
ObsDataVector<DATATYPE> handles vectors of data of type DATATYPE in observation space. More... | |
class | ObsDimInfo |
struct | ConvertType |
Template handlers for implicit variable conversion. More... | |
struct | ConvertType< double > |
class | ObsSpace |
Observation data class for IODA. More... | |
class | ObsFileInParameters |
class | ObsFileOutParameters |
class | ObsExtendParameters |
class | ObsGenerateParametersBase |
class | EmbeddedObsGenerateRandomParameters |
class | ObsGenerateRandomParameters |
Options controlling the ObsIoGenerateRandom class. More... | |
class | EmbeddedObsGenerateListParameters |
class | ObsGenerateListParameters |
Options controlling the ObsIoGenerateList class. More... | |
class | LegacyObsGenerateParameters |
Options in the 'generate' YAML section. More... | |
class | ObsIoParametersWrapper |
class | ObsTopLevelParameters |
class | ObsSpaceParameters |
class | ObsVector |
ObsVector class to handle vectors in observation space for IODA. More... | |
Typedefs | |
typedef std::vector< std::pair< std::string, Variable > > | VarNameObjectList |
typedef for holding list of variable names with associated variable object More... | |
typedef std::map< std::string, std::vector< std::string > > | VarDimMap |
typedef for holding dim names attached to variables More... | |
typedef std::vector< std::shared_ptr< NewDimensionScale_Base > > | NewDimensionScales_t |
template<typename DataType > | |
using | Object_Accessor = typename detail::Object_AccessorTypedef< DataType >::type |
typedef std::vector< Variable > | NewVariables_Scales_t |
typedef std::vector< std::shared_ptr< NewVariable_Base > > | NewVariables_t |
template<typename DATATYPE > | |
using | ObsDataRow = std::vector< DATATYPE > |
Enumerations | |
enum class | FileFormat { AUTO , HDF5 , ODB } |
Observation file format. More... | |
enum class | ioda_Eigen_Resize { Resize , No_Resize } |
Do we want to auto-resize the Eigen object on read to fit the data being read? More... | |
enum class | BasicTypes { undefined_ , float_ , double_ , ldouble_ , char_ , short_ , ushort_ , int_ , uint_ , lint_ , ulint_ , llint_ , ullint_ , int32_ , uint32_ , int16_ , uint16_ , int64_ , uint64_ , bool_ , str_ } |
enum class | FillValuePolicy { HDF5 , NETCDF4 } |
This option describes the default fill values that will be used if the user does not manually specify a fill value. More... | |
enum class | SelectionOperator { SET , OR , AND , XOR , NOT_B , NOT_A , APPEND , PREPEND } |
Selection enum. More... | |
enum class | SelectionState { ALL , NONE } |
The "default" for the selection. More... | |
enum class | ObsIoModes { READ , WRITE } |
enum class | ObsDtype { None , Float , Integer , String , DateTime } |
enum class | ObsDimensionId { Nlocs , Nchans } |
Functions | |
FileFormat | determineFileFormat (const std::string &filePath, FileFormat hint) |
Determine the format of an observation file. More... | |
std::vector< std::size_t > | CharShapeFromStringVector (const std::vector< std::string > &StringVector) |
std::vector< std::string > | CharArrayToStringVector (const char *CharData, const std::vector< std::size_t > &CharShape) |
void | StringVectorToCharArray (const std::vector< std::string > &StringVector, const std::vector< std::size_t > &CharShape, char *CharData) |
std::string | TypeIdName (const std::type_info &TypeId) |
std::size_t | FindMaxStringLength (const std::vector< std::string > &StringVector) |
std::string | fullVarName (const std::string &groupName, const std::string &varName) |
form full variable name given individual group and variable names More... | |
void | collectVarDimInfo (const ObsGroup &obsGroup, VarNameObjectList &varObjectList, VarNameObjectList &dimVarObjectList, VarDimMap &dimsAttachedToVars, Dimensions_t &maxVarSize0) |
collect variable and dimension information from a ioda ObsGroup More... | |
std::type_index | varDtype (const Group &group, const std::string &varName) |
get variable data type More... | |
bool | varIsDimScale (const Group &group, const std::string &varName) |
true if variable is a dimension scale More... | |
std::vector< util::DateTime > | convertDtStringsToDtime (const std::vector< std::string > &dtStrings) |
convert datetime strings to DateTime object More... | |
std::vector< util::DateTime > | convertRefOffsetToDtime (const int refIntDtime, const std::vector< float > &timeOffsets) |
convert reference, time to DateTime object More... | |
std::vector< std::string > | StringArrayToStringVector (const std::vector< std::string > &arrayData, const std::vector< Dimensions_t > &arrayShape) |
convert 2D string array to a vector of strings More... | |
void | setOfileParamsFromTestConfig (const eckit::LocalConfiguration &obsConfig, ioda::ObsSpaceParameters &obsParams) |
set params for output file construction from test YAML configuration More... | |
std::string | uniquifyFileName (const std::string &fileName, const std::size_t rankNum, const int timeRankNum) |
uniquify the output file name More... | |
std::string | convertNewVnameToOldVname (const std::string &varName) |
convert the new format varible name to the old format More... | |
VarDimMap | genDimsAttachedToVars (const Has_Variables &varContainer, const std::vector< std::string > &varList, const std::vector< std::string > &dimVarList) |
form a map containing lists of dimension variables that are attached to each variable More... | |
template<typename FromType , typename ToType > | |
void | ConvertVarType (const std::vector< FromType > &FromVar, std::vector< ToType > &ToVar) |
const ObsSpace * | obsspace_construct_f (const eckit::Configuration *conf, const util::DateTime *begin, const util::DateTime *end) |
void | obsspace_destruct_f (ObsSpace *obss) |
void | obsspace_obsname_f (const ObsSpace &obss, size_t &lcname, char *cname) |
const oops::Variables * | obsspace_obsvariables_f (const ObsSpace &obss) |
std::size_t | obsspace_get_gnlocs_f (const ObsSpace &obss) |
std::size_t | obsspace_get_nlocs_f (const ObsSpace &obss) |
std::size_t | obsspace_get_nchans_f (const ObsSpace &obss) |
std::size_t | obsspace_get_nrecs_f (const ObsSpace &obss) |
std::size_t | obsspace_get_nvars_f (const ObsSpace &obss) |
void | obsspace_get_dim_name_f (const ObsSpace &obss, const int &dim_id, std::size_t &len_dim_name, char *dim_name) |
std::size_t | obsspace_get_dim_size_f (const ObsSpace &obss, const int &dim_id) |
int | obsspace_get_dim_id_f (const ObsSpace &obss, const char *dim_name) |
void | obsspace_get_comm_f (const ObsSpace &obss, int &lcname, char *cname) |
void | obsspace_get_recnum_f (const ObsSpace &obss, const std::size_t &length, std::size_t *recnum) |
void | obsspace_get_index_f (const ObsSpace &obss, const std::size_t &length, std::size_t *index) |
bool | obsspace_has_f (const ObsSpace &obss, const char *group, const char *vname) |
void | obsspace_get_int32_f (const ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, int32_t *vec, const std::size_t &len_cs, int *chan_select) |
void | obsspace_get_int64_f (const ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, int64_t *vec, const std::size_t &len_cs, int *chan_select) |
void | obsspace_get_real32_f (const ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, float *vec, const std::size_t &len_cs, int *chan_select) |
void | obsspace_get_real64_f (const ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, double *vec, const std::size_t &len_cs, int *chan_select) |
void | obsspace_get_datetime_f (const ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, int32_t *date, int32_t *time, const std::size_t &len_cs, int *chan_select) |
void | obsspace_put_int32_f (ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, int32_t *vec, const std::size_t &ndims, int *dim_ids) |
void | obsspace_put_int64_f (ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, int64_t *vec, const std::size_t &ndims, int *dim_ids) |
void | obsspace_put_real32_f (ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, float *vec, const std::size_t &ndims, int *dim_ids) |
void | obsspace_put_real64_f (ObsSpace &obss, const char *group, const char *vname, const std::size_t &length, double *vec, const std::size_t &ndims, int *dim_ids) |
int | obsspace_get_nlocs_dim_id_f () |
int | obsspace_get_nchans_dim_id_f () |
double | dotProduct (const Distribution &dist, std::size_t numVariables, const std::vector< double > &v1, const std::vector< double > &v2) |
double | dotProduct (const Distribution &dist, std::size_t numVariables, const std::vector< float > &v1, const std::vector< float > &v2) |
double | dotProduct (const Distribution &dist, std::size_t numVariables, const std::vector< int > &v1, const std::vector< int > &v2) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, std::size_t numVariables, const std::vector< double > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, std::size_t numVariables, const std::vector< float > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, std::size_t numVariables, const std::vector< int > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, std::size_t numVariables, const std::vector< std::string > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, std::size_t numVariables, const std::vector< util::DateTime > &v) |
std::shared_ptr< Distribution > | createReplicaDistribution (const eckit::mpi::Comm &comm, std::shared_ptr< const Distribution > master, const std::vector< std::size_t > &masterRecordNums) |
Create a suitable replica distribution for the distribution master . More... | |
std::size_t | globalNumNonMissingObs (const Distribution &dist, size_t numVariables, const std::vector< float > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, size_t numVariables, const std::vector< int > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, size_t numVariables, const std::vector< std::string > &v) |
std::size_t | globalNumNonMissingObs (const Distribution &dist, size_t numVariables, const std::vector< util::DateTime > &v) |
IODA_DL void | copy (const ObjectSelection &from, ObjectSelection &to, const ScaleMapping &scale_map) |
Generic data copying function. More... | |
IODA_DL void | unwind_exception_stack (const std::exception &e, std::ostream &out=std::cerr, int level=0) |
Convenience function for unwinding an exception stack. More... | |
template<class DataType > | |
std::shared_ptr< NewDimensionScale_Object< DataType > > | NewDimensionScale (const std::string &name, Dimensions_t size, Dimensions_t maxSize=Unspecified, Dimensions_t chunkingSize=Unspecified) |
Wrapper function used when listing new dimension scales to construct. More... | |
template<class DataType > | |
std::shared_ptr< NewDimensionScale_Object< DataType > > | NewDimensionScale (const std::string &name, ScaleSizes sizes) |
IODA_DL std::shared_ptr< NewDimensionScale_Base > | NewDimensionScale (const std::string &name, const Type &t, Dimensions_t size, Dimensions_t maxSize=Unspecified, Dimensions_t chunkingSize=Unspecified) |
IODA_DL std::shared_ptr< NewDimensionScale_Base > | NewDimensionScale (const std::string &name, const Variable &scale, const ScaleSizes &overrides=ScaleSizes()) |
IODA_DL std::vector< std::string > | splitPaths (const std::string &p) |
Split a string based on occurances of the '/' character. More... | |
IODA_DL std::string | condensePaths (const std::vector< std::string > &p, size_t start=0, size_t end=std::string::npos) |
The inverse of splitPaths. Concatenate strings, separating with '/'. More... | |
IODA_DL std::vector< std::string > | concatenateStringVectors (const std::vector< std::vector< std::string >> &stringVectors) |
Concatenate equal-length vectors of strings element-by-element. Removes trailing spaces. More... | |
IODA_DL std::string | convertV1PathToV2Path (const std::string &path) |
Split path into substrings separated by @ characters, then concatenate them in reverse order, replacing the @ s with / s. More... | |
IODA_DL void | convertColumn (const std::string &unit, std::vector< double > &dataToConvert) |
IODA_DL std::string | getSIUnit (const std::string &unit) |
template<class DataType , bool FreeOnClose> | |
void | FreeType (DataType, typename std::enable_if<!std::is_pointer< DataType >::value >::type *=0) |
template<class DataType , bool FreeOnClose> | |
void | FreeType (DataType, typename std::enable_if< std::is_pointer< DataType >::value >::type *=0, typename std::enable_if<!FreeOnClose >::type *=0) |
template<class DataType , bool FreeOnClose> | |
void | FreeType (DataType d, typename std::enable_if< std::is_pointer< DataType >::value >::type *=0, typename std::enable_if< FreeOnClose >::type *=0) |
template<class DataType > | |
std::shared_ptr< NewVariable_Base > | NewVariable (const std::string &name, const NewVariables_Scales_t &scales, const VariableCreationParameters ¶ms=VariableCreationParameters::defaulted< DataType >()) |
std::shared_ptr< NewVariable_Base > | NewVariable (const std::string &name, const Type &DataType, const NewVariables_Scales_t &scales, const VariableCreationParameters ¶ms=VariableCreationParameters()) |
std::vector< std::string > | concatenateStringVectors (const std::vector< std::vector< std::string > > &stringVectors) |
void | storeGenData (const std::vector< float > &latVals, const std::vector< float > &lonVals, const std::vector< std::string > &dtStrings, const std::vector< std::string > &obsVarNames, const std::vector< float > &obsErrors, ObsGroup &obsGroup) |
store generated data into an ObsGroup More... | |
template<typename DATATYPE > | |
void | printNonnumericObsDataVectorStats (const ObsDataVector< DATATYPE > &obsdatavector, const ObsSpace &obsdb, std::ostream &os) |
template<typename DATATYPE > | |
void | printNumericObsDataVectorStats (const ObsDataVector< DATATYPE > &obsdatavector, const ObsSpace &obsdb, std::ostream &os) |
Variables | |
static DistributionMaker< AtlasDistribution > | maker (DIST_NAME) |
static DistributionMaker< Halo > | maker ("Halo") |
static DistributionMaker< InefficientDistribution > | maker ("InefficientDistribution") |
static DistributionMaker< RoundRobin > | maker (DIST_NAME) |
constexpr int | Unlimited = -1 |
Specifies that a dimension is resizable to infinity. More... | |
constexpr int | Unspecified = -2 |
Specifies that a dimension has no specified size. Tells ioda to figure it out from elsewhere. More... | |
static ObsIoMaker< ObsIoFileCreate > | maker ("FileCreate") |
static ObsIoMaker< ObsIoFileRead > | maker ("FileRead") |
static ObsIoMaker< ObsIoGenerateList > | maker ("GenerateList") |
static ObsIoMaker< ObsIoGenerateRandom > | maker ("GenerateRandom") |
constexpr int | DEFAULT_FRAME_SIZE = 10000 |
typedef std::vector<std::shared_ptr<NewDimensionScale_Base> > ioda::NewDimensionScales_t |
Definition at line 89 of file DimensionScales.h.
typedef std::vector<Variable> ioda::NewVariables_Scales_t |
Definition at line 147 of file Has_Variables.h.
typedef std::vector<std::shared_ptr<NewVariable_Base> > ioda::NewVariables_t |
Definition at line 178 of file Has_Variables.h.
using ioda::ObsDataRow = typedef std::vector<DATATYPE> |
Definition at line 37 of file src/ObsDataVector.h.
typedef std::map<std::string, std::vector<std::string> > ioda::VarDimMap |
typedef for holding dim names attached to variables
Definition at line 36 of file IodaUtils.h.
typedef std::vector<std::pair<std::string, Variable> > ioda::VarNameObjectList |
typedef for holding list of variable names with associated variable object
Definition at line 33 of file IodaUtils.h.
|
strong |
Basic pre-defined types (Python convenience wrappers)
Enumerator | |
---|---|
undefined_ | Internal use only. |
float_ | |
double_ | |
ldouble_ | |
char_ | |
short_ | |
ushort_ | |
int_ | |
uint_ | |
lint_ | |
ulint_ | |
llint_ | |
ullint_ | |
int32_ | |
uint32_ | |
int16_ | |
uint16_ | |
int64_ | |
uint64_ | |
bool_ | |
str_ |
|
strong |
Observation file format.
Enumerator | |
---|---|
AUTO | File format determined automatically from the file name extension ( |
HDF5 | HDF5 file format. |
ODB | ODB file format. |
Definition at line 19 of file src/core/FileFormat.h.
|
strong |
Do we want to auto-resize the Eigen object on read to fit the data being read?
Enumerator | |
---|---|
Resize | Yes. |
No_Resize | No. |
Definition at line 41 of file Eigen_Compat.h.
|
strong |
Enumerator | |
---|---|
Nlocs | |
Nchans |
Definition at line 61 of file src/ObsSpace.h.
|
strong |
Enumerator | |
---|---|
None | |
Float | |
Integer | |
String |
Definition at line 49 of file src/ObsSpace.h.
|
strong |
Enumerator | |
---|---|
READ | |
WRITE |
Definition at line 24 of file ObsIoFactory.h.
std::vector< std::string > ioda::CharArrayToStringVector | ( | const char * | CharData, |
const std::vector< std::size_t > & | CharShape | ||
) |
Definition at line 36 of file IodaUtils.cc.
std::vector< std::size_t > ioda::CharShapeFromStringVector | ( | const std::vector< std::string > & | StringVector | ) |
Definition at line 20 of file IodaUtils.cc.
void ioda::collectVarDimInfo | ( | const ObsGroup & | obsGroup, |
VarNameObjectList & | varObjectList, | ||
VarNameObjectList & | dimVarObjectList, | ||
VarDimMap & | dimsAttachedToVars, | ||
Dimensions_t & | maxVarSize0 | ||
) |
collect variable and dimension information from a ioda ObsGroup
It is assumed that the input ObsGroup has been populated. For example you open an existing hdf5 file, and then call this routing to collect the information. The information collected is passed back through the output parameters (last 4 parameters) of this routine.
The reason for collecting all of this information in a single routine is to handle severe performance issues with the HDF5 library when inspecting an ObsGroup based on an HDF5 backend.
obsGroup | ioda ObsGroup object |
varObjectList | list of regular variable names with associated Variable objects |
dimVarObjectList | list of dimension variable names with associated Variable objects |
dimsAttachedToVars | map structure holding list of dimension scale names attached each regular variable |
maxVarSize0 | maximum var length along the first (0th) dimension |
Definition at line 125 of file IodaUtils.cc.
std::vector<std::string> ioda::concatenateStringVectors | ( | const std::vector< std::vector< std::string > > & | stringVectors | ) |
Definition at line 57 of file StringFuncs.cpp.
IODA_DL std::vector<std::string> ioda::concatenateStringVectors | ( | const std::vector< std::vector< std::string >> & | stringVectors | ) |
Concatenate equal-length vectors of strings element-by-element. Removes trailing spaces.
stringVectors | represents the vectors of strings all stored together in one vector. |
void ioda::convertColumn | ( | const std::string & | unit, |
std::vector< double > & | dataToConvert | ||
) |
Definition at line 16 of file UnitConversions.cpp.
std::vector<util::DateTime> ioda::convertDtStringsToDtime | ( | const std::vector< std::string > & | dtStrings | ) |
convert datetime strings to DateTime object
dtStrings | datetime strings |
Definition at line 239 of file IodaUtils.cc.
std::string ioda::convertNewVnameToOldVname | ( | const std::string & | varName | ) |
convert the new format varible name to the old format
varName | new format variable name |
Definition at line 365 of file IodaUtils.cc.
std::vector<util::DateTime> ioda::convertRefOffsetToDtime | ( | const int | refIntDtime, |
const std::vector< float > & | timeOffsets | ||
) |
convert reference, time to DateTime object
refDtime | reference date time |
timeOffets | offset time values (in hours) |
Definition at line 251 of file IodaUtils.cc.
std::string ioda::convertV1PathToV2Path | ( | const std::string & | path | ) |
Split path
into substrings separated by @
characters, then concatenate them in reverse order, replacing the @
s with /
s.
For example, longitude@MetaData becomes MetaData/longitude.
path | is the path to convert |
Definition at line 85 of file StringFuncs.cpp.
void ioda::ConvertVarType | ( | const std::vector< FromType > & | FromVar, |
std::vector< ToType > & | ToVar | ||
) |
This method will perform numeric data type conversions. This method is aware of the IODA missing values and will convert these appropriately. For example when converting double to float, all double missing values will be replaced with float missing values during the conversion.
[in] | FromVar | Vector of variable we are converting from |
[out] | ToVar | Vector of variable we are converting to |
[in] | VarSize | Total number of elements in FromVar and ToVar. |
Definition at line 148 of file IodaUtils.h.
void ioda::copy | ( | const ObjectSelection & | from, |
ObjectSelection & | to, | ||
const ScaleMapping & | scale_map | ||
) |
Generic data copying function.
from | contains the objects that we are copying |
to | contains the destination(s) of the copy |
scale_map | contains settings regarding how dimension scales are propagated |
Definition at line 63 of file Copying.cpp.
std::shared_ptr<Distribution> ioda::createReplicaDistribution | ( | const eckit::mpi::Comm & | comm, |
std::shared_ptr< const Distribution > | master, | ||
const std::vector< std::size_t > & | masterRecordNums | ||
) |
Create a suitable replica distribution for the distribution master
.
A replica distribution assigns each record r
to a process if and only if another distribution (the master distribution) has assigned the same record to that process.
comm | Communicator used by the master distribution. |
master | Master distribution. |
masterRecordNums | Records of all observations assigned by the master distribution to the calling process. |
Definition at line 121 of file DistributionUtils.cc.
FileFormat ioda::determineFileFormat | ( | const std::string & | filePath, |
FileFormat | hint | ||
) |
Determine the format of an observation file.
Returns hint
unless it's set to FileFormat::AUTO, in which case the function returns FileFormat::ODB if filePath
ends with .odb
(irrespective of case) and FileFormat::HDF5 otherwise.
Definition at line 13 of file FileFormat.cc.
|
related |
double ioda::dotProduct | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< float > & | v1, | ||
const std::vector< float > & | v2 | ||
) |
double ioda::dotProduct | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< int > & | v1, | ||
const std::vector< int > & | v2 | ||
) |
std::size_t ioda::FindMaxStringLength | ( | const std::vector< std::string > & | StringVector | ) |
Definition at line 109 of file IodaUtils.cc.
void ioda::FreeType | ( | DataType | d, |
typename std::enable_if< std::is_pointer< DataType >::value >::type * | = 0 , |
||
typename std::enable_if< FreeOnClose >::type * | = 0 |
||
) |
Definition at line 37 of file Marshalling.h.
void ioda::FreeType | ( | DataType | , |
typename std::enable_if< std::is_pointer< DataType >::value >::type * | = 0 , |
||
typename std::enable_if<!FreeOnClose >::type * | = 0 |
||
) |
Definition at line 34 of file Marshalling.h.
std::string ioda::fullVarName | ( | const std::string & | groupName, |
const std::string & | varName | ||
) |
form full variable name given individual group and variable names
This routine will form the consistent frontend variable name according
groupName | name of group |
varName | name of variable |
Definition at line 120 of file IodaUtils.cc.
VarDimMap ioda::genDimsAttachedToVars | ( | const Has_Variables & | varContainer, |
const std::vector< std::string > & | varList, | ||
const std::vector< std::string > & | dimVarList | ||
) |
form a map containing lists of dimension variables that are attached to each variable
varContainer | Has_Variables object with variables to check |
varList | list of regular variables |
dimVarList | list of dimension scale variables |
std::string ioda::getSIUnit | ( | const std::string & | unit | ) |
Definition at line 27 of file UnitConversions.cpp.
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
size_t | numVariables, | ||
const std::vector< float > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
size_t | numVariables, | ||
const std::vector< int > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
size_t | numVariables, | ||
const std::vector< std::string > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
size_t | numVariables, | ||
const std::vector< util::DateTime > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< double > & | v | ||
) |
Definition at line 90 of file DistributionUtils.cc.
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< float > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< int > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< std::string > & | v | ||
) |
std::size_t ioda::globalNumNonMissingObs | ( | const Distribution & | dist, |
std::size_t | numVariables, | ||
const std::vector< util::DateTime > & | v | ||
) |
std::shared_ptr< NewDimensionScale_Base > ioda::NewDimensionScale | ( | const std::string & | name, |
const Type & | t, | ||
Dimensions_t | size, | ||
Dimensions_t | maxSize = Unspecified , |
||
Dimensions_t | chunkingSize = Unspecified |
||
) |
Definition at line 17 of file DimensionScales.cpp.
std::shared_ptr< NewDimensionScale_Base > ioda::NewDimensionScale | ( | const std::string & | name, |
const Variable & | scale, | ||
const ScaleSizes & | overrides = ScaleSizes() |
||
) |
|
inline |
Wrapper function used when listing new dimension scales to construct.
DataType | is the type of data used in the scale. |
name | is the new scale's name. |
size | is the initial size (in elements). |
maxSize | is the maximum size, in elements. ioda::Unspecified sets the max dimension size to the initial size. ioda::Unlimited specifies an unlimited dimension, which should be seldom used. |
chunkingSize | is a "hint" parameter that guides how data are grouped in memory. New variables using this scale take this hint to specify their chunk sizes. |
Definition at line 125 of file DimensionScales.h.
|
inline |
Definition at line 134 of file DimensionScales.h.
|
inline |
|
inline |
Definition at line 187 of file Has_Variables.h.
const ObsSpace * ioda::obsspace_construct_f | ( | const eckit::Configuration * | conf, |
const util::DateTime * | begin, | ||
const util::DateTime * | end | ||
) |
Definition at line 24 of file obsspace_f.cc.
void ioda::obsspace_destruct_f | ( | ObsSpace * | obss | ) |
Definition at line 33 of file obsspace_f.cc.
void ioda::obsspace_get_comm_f | ( | const ObsSpace & | obss, |
int & | lcname, | ||
char * | cname | ||
) |
Definition at line 90 of file obsspace_f.cc.
void ioda::obsspace_get_datetime_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
int32_t * | date, | ||
int32_t * | time, | ||
const std::size_t & | len_cs, | ||
int * | chan_select | ||
) |
Definition at line 170 of file obsspace_f.cc.
int ioda::obsspace_get_dim_id_f | ( | const ObsSpace & | obss, |
const char * | dim_name | ||
) |
Definition at line 85 of file obsspace_f.cc.
void ioda::obsspace_get_dim_name_f | ( | const ObsSpace & | obss, |
const int & | dim_id, | ||
std::size_t & | len_dim_name, | ||
char * | dim_name | ||
) |
Definition at line 73 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_dim_size_f | ( | const ObsSpace & | obss, |
const int & | dim_id | ||
) |
Definition at line 81 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_gnlocs_f | ( | const ObsSpace & | obss | ) |
Definition at line 52 of file obsspace_f.cc.
void ioda::obsspace_get_index_f | ( | const ObsSpace & | obss, |
const std::size_t & | length, | ||
std::size_t * | index | ||
) |
Definition at line 104 of file obsspace_f.cc.
void ioda::obsspace_get_int32_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
int32_t * | vec, | ||
const std::size_t & | len_cs, | ||
int * | chan_select | ||
) |
void ioda::obsspace_get_int64_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
int64_t * | vec, | ||
const std::size_t & | len_cs, | ||
int * | chan_select | ||
) |
Definition at line 132 of file obsspace_f.cc.
int ioda::obsspace_get_nchans_dim_id_f | ( | ) |
Definition at line 291 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_nchans_f | ( | const ObsSpace & | obss | ) |
Definition at line 60 of file obsspace_f.cc.
int ioda::obsspace_get_nlocs_dim_id_f | ( | ) |
Definition at line 287 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_nlocs_f | ( | const ObsSpace & | obss | ) |
Definition at line 56 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_nrecs_f | ( | const ObsSpace & | obss | ) |
Definition at line 64 of file obsspace_f.cc.
std::size_t ioda::obsspace_get_nvars_f | ( | const ObsSpace & | obss | ) |
Definition at line 68 of file obsspace_f.cc.
void ioda::obsspace_get_real32_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
float * | vec, | ||
const std::size_t & | len_cs, | ||
int * | chan_select | ||
) |
Definition at line 144 of file obsspace_f.cc.
void ioda::obsspace_get_real64_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
double * | vec, | ||
const std::size_t & | len_cs, | ||
int * | chan_select | ||
) |
void ioda::obsspace_get_recnum_f | ( | const ObsSpace & | obss, |
const std::size_t & | length, | ||
std::size_t * | recnum | ||
) |
Definition at line 96 of file obsspace_f.cc.
bool ioda::obsspace_has_f | ( | const ObsSpace & | obss, |
const char * | group, | ||
const char * | vname | ||
) |
Definition at line 114 of file obsspace_f.cc.
void ioda::obsspace_obsname_f | ( | const ObsSpace & | obss, |
size_t & | lcname, | ||
char * | cname | ||
) |
Definition at line 39 of file obsspace_f.cc.
const oops::Variables * ioda::obsspace_obsvariables_f | ( | const ObsSpace & | obss | ) |
Definition at line 47 of file obsspace_f.cc.
void ioda::obsspace_put_int32_f | ( | ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
int32_t * | vec, | ||
const std::size_t & | ndims, | ||
int * | dim_ids | ||
) |
Definition at line 201 of file obsspace_f.cc.
void ioda::obsspace_put_int64_f | ( | ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
int64_t * | vec, | ||
const std::size_t & | ndims, | ||
int * | dim_ids | ||
) |
Definition at line 224 of file obsspace_f.cc.
void ioda::obsspace_put_real32_f | ( | ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
float * | vec, | ||
const std::size_t & | ndims, | ||
int * | dim_ids | ||
) |
Definition at line 244 of file obsspace_f.cc.
void ioda::obsspace_put_real64_f | ( | ObsSpace & | obss, |
const char * | group, | ||
const char * | vname, | ||
const std::size_t & | length, | ||
double * | vec, | ||
const std::size_t & | ndims, | ||
int * | dim_ids | ||
) |
Definition at line 264 of file obsspace_f.cc.
void ioda::printNonnumericObsDataVectorStats | ( | const ObsDataVector< DATATYPE > & | obsdatavector, |
const ObsSpace & | obsdb, | ||
std::ostream & | os | ||
) |
Print statistics describing a vector obsdatavector
of observations taken from obsdb
to the stream os
.
This is an implementation suitable for non-numeric data. Users shouldn't need to call it directly; they should call ObsDataVector::print() instead.
Definition at line 227 of file src/ObsDataVector.h.
void ioda::printNumericObsDataVectorStats | ( | const ObsDataVector< DATATYPE > & | obsdatavector, |
const ObsSpace & | obsdb, | ||
std::ostream & | os | ||
) |
Print statistics describing a vector obsdatavector
of observations taken from obsdb
to the stream os
.
This is an implementation suitable for numeric data. Users shouldn't need to call it directly; they should call ObsDataVector::print() instead.
Definition at line 249 of file src/ObsDataVector.h.
void ioda::setOfileParamsFromTestConfig | ( | const eckit::LocalConfiguration & | obsConfig, |
ioda::ObsSpaceParameters & | obsParams | ||
) |
set params for output file construction from test YAML configuration
This routine is intended for use by the ObsIo and ObsFrame tests. It relies on test confiruation "test data.write dimensions" and "test data.write variables" accompanying an "obsdataout.obsfile" spec in the YAML configuration.
obsConfig | test YAML configuration for and ObsSpace |
obsParams | output params |
Definition at line 306 of file IodaUtils.cc.
void ioda::storeGenData | ( | const std::vector< float > & | latVals, |
const std::vector< float > & | lonVals, | ||
const std::vector< std::string > & | dtStrings, | ||
const std::vector< std::string > & | obsVarNames, | ||
const std::vector< float > & | obsErrors, | ||
ObsGroup & | obsGroup | ||
) |
store generated data into an ObsGroup
latVals | vector of latitude values | |
lonVals | vector of longitude values | |
dtStrings | vector of datetime (ISO 8601 string) values | |
obsVarNames | vector (string) of simulated variable names | |
obsErrors | vector of obs error estimates | |
[out] | obsGroup | destination for the generated data |
Definition at line 18 of file ObsIoGenerateUtils.cc.
std::vector<std::string> ioda::StringArrayToStringVector | ( | const std::vector< std::string > & | arrayData, |
const std::vector< Dimensions_t > & | arrayShape | ||
) |
convert 2D string array to a vector of strings
The incoming 2D strings array is passed in through the arrayData argument as a flattened vector. The arrayShape arugument shows how to un-flatten the arrayData.
arrayData | Flattened data for input 2D string array |
arrayShape | Shape of input arrayData |
Definition at line 274 of file IodaUtils.cc.
void ioda::StringVectorToCharArray | ( | const std::vector< std::string > & | StringVector, |
const std::vector< std::size_t > & | CharShape, | ||
char * | CharData | ||
) |
Definition at line 70 of file IodaUtils.cc.
std::string ioda::TypeIdName | ( | const std::type_info & | TypeId | ) |
std::string ioda::uniquifyFileName | ( | const std::string & | fileName, |
const std::size_t | rankNum, | ||
const int | timeRankNum | ||
) |
uniquify the output file name
This function will tag on the MPI task number to the end of the file name to avoid collisions when running with multiple MPI tasks.
fileName | raw output file name |
rankNum | MPI group communicator rank number |
timeRankNum | MPI time communicator rank number |
Definition at line 343 of file IodaUtils.cc.
void ioda::unwind_exception_stack | ( | const std::exception & | e, |
std::ostream & | out = std::cerr , |
||
int | level = 0 |
||
) |
Convenience function for unwinding an exception stack.
Definition at line 48 of file Exception.cpp.
std::type_index ioda::varDtype | ( | const Group & | group, |
const std::string & | varName | ||
) |
get variable data type
Definition at line 221 of file IodaUtils.cc.
bool ioda::varIsDimScale | ( | const Group & | group, |
const std::string & | varName | ||
) |
true if variable is a dimension scale
Definition at line 233 of file IodaUtils.cc.
|
constexpr |
Definition at line 20 of file ObsIoParametersBase.h.
|
static |
|
static |
|
static |
|
static |
|
static |
|
static |
|
static |
|
static |
|
constexpr |
Specifies that a dimension is resizable to infinity.
Definition at line 25 of file DimensionScales.h.
|
constexpr |
Specifies that a dimension has no specified size. Tells ioda to figure it out from elsewhere.
Definition at line 27 of file DimensionScales.h.