This class provides access to observations that may be held on multiple MPI ranks. More...

#include <ObsAccessor.h>

Public Member Functions
	~ObsAccessor ()=default

	ObsAccessor (const ObsAccessor &)=delete

	ObsAccessor (ObsAccessor &&)=default

ObsAccessor &	operator= (const ObsAccessor &)=delete

ObsAccessor &	operator= (ObsAccessor &&)=default

std::vector< size_t >	getValidObservationIds (const std::vector< bool > &apply, const ioda::ObsDataVector< int > &flags, const Variables &filtervars, bool validIfAnyFilterVariablePassedQC=true) const
	Return the IDs of observation locations that should be treated as valid by a filter. More...

std::vector< size_t >	getValidObservationIds (const std::vector< bool > &apply) const
	Return the IDs of both flagged and unflagged observation locations selected by the where clause. More...

std::vector< int >	getIntVariableFromObsSpace (const std::string &group, const std::string &variable) const
	Return the values of the specified variable at successive observation locations. More...

std::vector< float >	getFloatVariableFromObsSpace (const std::string &group, const std::string &variable) const

std::vector< double >	getDoubleVariableFromObsSpace (const std::string &group, const std::string &variable) const

std::vector< std::string >	getStringVariableFromObsSpace (const std::string &group, const std::string &variable) const

std::vector< util::DateTime >	getDateTimeVariableFromObsSpace (const std::string &group, const std::string &variable) const

std::vector< size_t >	getRecordIds () const
	Return the vector of IDs of records successive observation locations belong to. More...

size_t	totalNumObservations () const

RecursiveSplitter	splitObservationsIntoIndependentGroups (const std::vector< size_t > &validObsIds, bool opsCompatibilityMode=false) const

void	flagRejectedObservations (const std::vector< bool > &isRejected, std::vector< std::vector< bool > > &flagged) const
	Update flags of observations held on the current MPI rank. More...

Static Public Member Functions
static ObsAccessor	toAllObservations (const ioda::ObsSpace &obsdb)
	Create an accessor to observations from the observation space `obsdb`, assuming that the whole set of observations held on all MPI ranks must be processed together as a single group. More...

static ObsAccessor	toObservationsSplitIntoIndependentGroupsByRecordId (const ioda::ObsSpace &obsdb)
	Create an accessor to the collection of observations held in `obsdb`, assuming that each record can be processed independently. More...

static ObsAccessor	toObservationsSplitIntoIndependentGroupsByVariable (const ioda::ObsSpace &obsdb, const Variable &variable)
	Create an accessor to the collection of observations held in `obsdb`, assuming that observations with different values of the variable `variable` can be processed independently. More...

Private Types
enum class	GroupBy { NOTHING , RECORD_ID , VARIABLE }

Private Member Functions
	ObsAccessor (const ioda::ObsSpace &obsdb, GroupBy groupBy, boost::optional< Variable > categoryVariable)

bool	wereRecordsGroupedByCategoryVariable () const

void	groupObservationsByRecordNumber (const std::vector< size_t > &validObsIds, RecursiveSplitter &splitter) const

void	groupObservationsByCategoryVariable (const std::vector< size_t > &validObsIds, RecursiveSplitter &splitter) const

bool	isValid (const std::vector< ioda::ObsDataRow< int >> &flags, size_t ObsId, bool validIfAnyFilterVariablePassedQC) const
	Return true if filtered variable(s) have passed QC, otherwise false. More...

Private Attributes
const ioda::ObsSpace *	obsdb_

std::shared_ptr< const ioda::Distribution >	obsDistribution_

GroupBy	groupBy_

boost::optional< Variable >	categoryVariable_

Detailed Description

This class provides access to observations that may be held on multiple MPI ranks.

It is used by filters that may be configured to

process observations held on all MPI ranks as a single group
process observations from each record (by definition, held on a single MPI rank) independently from all others
process observations with each distinct value of a particular variable (held on a single MPI rank if this variable was used to group observations into records or on multiple MPI ranks if not) independently from all others.

Depending on which of these cases applies, create an ObservationAccessor object by calling the ObsAccessor::toAllObservations(), ObsAccessor::toObservationsSplitIntoIndependentGroupsByRecordId() or the ObsAccessor::toObservationsSplitIntoIndependentGroupsByVariable() static function. The ObsAccessor will then determine whether each independent group consists of observations held only on a single MPI rank. If so, methods such as getValidObservationIds() and getIntVariableFromObsSpace() will return vectors constructed from data held only on the current MPI rank (without any MPI communication); otherwise, these vectors will be constructed from data obtained from all MPI ranks.

Call splitObservationsIntoIndependentGroups() to construct a RecursiveSplitter object whose groups() method will return groups of observations that can be processed independently from each other (according to the criterion specified when the ObsAccessor was constructed).

Definition at line 56 of file ObsAccessor.h.

Member Enumeration Documentation

◆ GroupBy

enum ufo::ObsAccessor::GroupBy

strongprivate

Enumerator
NOTHING
RECORD_ID
VARIABLE

Definition at line 190 of file ObsAccessor.h.

Constructor & Destructor Documentation

◆ ~ObsAccessor()

ufo::ObsAccessor::~ObsAccessor ( )

default

◆ ObsAccessor() [1/3]

ufo::ObsAccessor::ObsAccessor ( const ObsAccessor & )

delete

Here is the caller graph for this function:

◆ ObsAccessor() [2/3]

ufo::ObsAccessor::ObsAccessor ( ObsAccessor && )

default

◆ ObsAccessor() [3/3]

ufo::ObsAccessor::ObsAccessor	(	const ioda::ObsSpace &	obsdb,
		GroupBy	groupBy,
		boost::optional< Variable >	categoryVariable
	)

private

Private constructor. Construct instances of this class by calling toAllObservations(), toObservationsSplitIntoIndependentGroupsByRecordId() or toObservationsSplitIntoIndependentGroupsByVariable() instead.

Definition at line 63 of file ObsAccessor.cc.

Here is the call graph for this function:

Member Function Documentation

◆ flagRejectedObservations()

void ufo::ObsAccessor::flagRejectedObservations	(	const std::vector< bool > &	isRejected,
		std::vector< std::vector< bool > > &	flagged
	)		const

Update flags of observations held on the current MPI rank.

Parameters

	isRejected	A vector of length totalNumObservations() whose ith element indicates if ith observation should be rejected.
[in,out]	flagged	A vector of vectors, each with as many elements as there are observation locations on the current MPI rank. On output, flagged[i][j] will be set to true for each i if the element of isRejected corresponding to jth observation location on the current rank is true.

Definition at line 243 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getDateTimeVariableFromObsSpace()

std::vector< util::DateTime > ufo::ObsAccessor::getDateTimeVariableFromObsSpace	(	const std::string &	group,
		const std::string &	variable
	)		const

Definition at line 159 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getDoubleVariableFromObsSpace()

std::vector< double > ufo::ObsAccessor::getDoubleVariableFromObsSpace	(	const std::string &	group,
		const std::string &	variable
	)		const

Definition at line 149 of file ObsAccessor.cc.

◆ getFloatVariableFromObsSpace()

std::vector< float > ufo::ObsAccessor::getFloatVariableFromObsSpace	(	const std::string &	group,
		const std::string &	variable
	)		const

Definition at line 144 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getIntVariableFromObsSpace()

std::vector< int > ufo::ObsAccessor::getIntVariableFromObsSpace	(	const std::string &	group,
		const std::string &	variable
	)		const

Return the values of the specified variable at successive observation locations.

If each independent group of observations is stored entirely on a single MPI rank, the returned vector contains values observed at locations held on the current rank only. Otherwise the vector is a concatenation of vectors obtained on all ranks.

Definition at line 139 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getRecordIds()

std::vector< size_t > ufo::ObsAccessor::getRecordIds ( ) const

Return the vector of IDs of records successive observation locations belong to.

If each independent group of observations is stored entirely on a single MPI rank, the returned vector contains record IDs of observation locations held on the current rank only. Otherwise the vector is a concatenation of vectors obtained on all ranks.

Definition at line 164 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getStringVariableFromObsSpace()

std::vector< std::string > ufo::ObsAccessor::getStringVariableFromObsSpace	(	const std::string &	group,
		const std::string &	variable
	)		const

Definition at line 154 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ getValidObservationIds() [1/2]

std::vector< size_t > ufo::ObsAccessor::getValidObservationIds ( const std::vector< bool > & apply ) const

Return the IDs of both flagged and unflagged observation locations selected by the where clause.

Parameters

apply Vector whose ith element is set to true if ith observation location held on the current MPI rank was selected by the where clause in the filter's configuration.

An observation location is treated as valid if it has been selected by the where clause.

If each independent group of observations is stored entirely on a single MPI rank, the returned vector contains local IDs of valid observation locations held on the current rank only. Otherwise the vector contains global IDs of valid locations held on all ranks, with IDs from 0 to nlocs(0) - 1 corresponding to locations held on rank 0, IDs from nlocs(0) to nlocs(0) + nlocs(1) - 1 corresponding to locations held on rank 1 and so on, where nlocs(i) denotes the number of locations held on ith rank.

Definition at line 125 of file ObsAccessor.cc.

◆ getValidObservationIds() [2/2]

std::vector< size_t > ufo::ObsAccessor::getValidObservationIds	(	const std::vector< bool > &	apply,
		const ioda::ObsDataVector< int > &	flags,
		const Variables &	filtervars,
		bool	validIfAnyFilterVariablePassedQC = `true`
	)		const

Return the IDs of observation locations that should be treated as valid by a filter.

Parameters

apply	Vector whose ith element is set to true if ith observation location held on the current MPI rank was selected by the `where` clause in the filter's configuration.
flags	An ObsDataVector holding the QC flags (set by any filters run previously) of observations held on the current MPI rank.
filtervars	List of filter variables.
validIfAnyFilterVariablePassedQC	Boolean switch to treat an observation as valid if any filter variable has not been rejected. By default this is true; if false, the observation is only treated as valid if all filter variables have passed QC.

An observation location is treated as valid if (a) it has been selected by the where clause and (b) its QC flag(s) for (some/all) filtered variable(s) are set to pass (see below).

If each independent group of observations is stored entirely on a single MPI rank, the returned vector contains local IDs of valid observation locations held on the current rank only. Otherwise the vector contains global IDs of valid locations held on all ranks, with IDs from 0 to nlocs(0) - 1 corresponding to locations held on rank 0, IDs from nlocs(0) to nlocs(0) + nlocs(1) - 1 corresponding to locations held on rank 1 and so on, where nlocs(i) denotes the number of locations held on ith rank.

If there is more than one filtered variable, and their QC flags differ, there is a choice as to whether to treat observation locations as valid (i) where none of the filtered variables have so far been rejected, or (ii) where at least one of these variables has not yet been rejected. The latter choice (ii) is the default, configurable via the switch validIfAnyFilterVariablePassedQC.

Definition at line 99 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ groupObservationsByCategoryVariable()

void ufo::ObsAccessor::groupObservationsByCategoryVariable	(	const std::vector< size_t > &	validObsIds,
		RecursiveSplitter &	splitter
	)		const

private

Definition at line 222 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ groupObservationsByRecordNumber()

void ufo::ObsAccessor::groupObservationsByRecordNumber	(	const std::vector< size_t > &	validObsIds,
		RecursiveSplitter &	splitter
	)		const

private

Definition at line 214 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ isValid()

bool ufo::ObsAccessor::isValid	(	const std::vector< ioda::ObsDataRow< int >> &	flags,
		size_t	ObsId,
		bool	validIfAnyFilterVariablePassedQC
	)		const

private

Return true if filtered variable(s) have passed QC, otherwise false.

Parameters

flags	A vector of type ObsDataRow holding the QC flags for the subset of simulated variables present in the list of filtered variables.
ObsId	Index of observation location.
validIfAnyFilterVariablePassedQC	Boolean variable to decide how to treat observation locations where QC flags of filtered variables differ. If true, consider that observation has passed QC if any filtered variable has passed QC. If false, consider that observation has passed QC only if all filtered variables passed QC.

Definition at line 174 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ operator=() [1/2]

ObsAccessor& ufo::ObsAccessor::operator= ( const ObsAccessor & )

delete

◆ operator=() [2/2]

ObsAccessor& ufo::ObsAccessor::operator= ( ObsAccessor && )

default

◆ splitObservationsIntoIndependentGroups()

RecursiveSplitter ufo::ObsAccessor::splitObservationsIntoIndependentGroups	(	const std::vector< size_t > &	validObsIds,
		bool	opsCompatibilityMode = `false`
	)		const

Construct a RecursiveSplitter object whose groups() method will return groups of observations that can be processed independently from each other (according to the criterion specified when the ObsAccessor was constructed).

Parameters

validObsIds	Indices of valid observations.
opsCompatibilityMode	Parameter to pass to the RecursiveSplitter's constructor.

Definition at line 197 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ toAllObservations()

ObsAccessor ufo::ObsAccessor::toAllObservations ( const ioda::ObsSpace & obsdb )

static

Create an accessor to observations from the observation space obsdb, assuming that the whole set of observations held on all MPI ranks must be processed together as a single group.

Definition at line 84 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ toObservationsSplitIntoIndependentGroupsByRecordId()

ObsAccessor ufo::ObsAccessor::toObservationsSplitIntoIndependentGroupsByRecordId ( const ioda::ObsSpace & obsdb )

static

Create an accessor to the collection of observations held in obsdb, assuming that each record can be processed independently.

Definition at line 89 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ toObservationsSplitIntoIndependentGroupsByVariable()

ObsAccessor ufo::ObsAccessor::toObservationsSplitIntoIndependentGroupsByVariable	(	const ioda::ObsSpace &	obsdb,
		const Variable &	variable
	)

static

Create an accessor to the collection of observations held in obsdb, assuming that observations with different values of the variable variable can be processed independently.

Definition at line 94 of file ObsAccessor.cc.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ totalNumObservations()

size_t ufo::ObsAccessor::totalNumObservations ( ) const

If each independent group of observations is stored entirely on a single MPI rank, return the number of observation locations held on the current rank. Otherwise return the total number of observation locations held on all ranks.

Definition at line 170 of file ObsAccessor.cc.

Here is the caller graph for this function:

◆ wereRecordsGroupedByCategoryVariable()

bool ufo::ObsAccessor::wereRecordsGroupedByCategoryVariable ( ) const

private

Definition at line 259 of file ObsAccessor.cc.

Here is the caller graph for this function:

Member Data Documentation

◆ categoryVariable_

boost::optional<Variable> ufo::ObsAccessor::categoryVariable_

private

Definition at line 229 of file ObsAccessor.h.

◆ groupBy_

GroupBy ufo::ObsAccessor::groupBy_

private

Definition at line 228 of file ObsAccessor.h.

◆ obsdb_

const ioda::ObsSpace* ufo::ObsAccessor::obsdb_

private

Definition at line 225 of file ObsAccessor.h.

◆ obsDistribution_

std::shared_ptr<const ioda::Distribution> ufo::ObsAccessor::obsDistribution_

private

Definition at line 226 of file ObsAccessor.h.

The documentation for this class was generated from the following files:

Public Member Functions

Static Public Member Functions

Private Types

Private Member Functions

Private Attributes

Detailed Description

Member Enumeration Documentation

◆ GroupBy

Constructor & Destructor Documentation

◆ ~ObsAccessor()

◆ ObsAccessor() [1/3]

◆ ObsAccessor() [2/3]

◆ ObsAccessor() [3/3]

Member Function Documentation

◆ flagRejectedObservations()

◆ getDateTimeVariableFromObsSpace()

◆ getDoubleVariableFromObsSpace()

◆ getFloatVariableFromObsSpace()

◆ getIntVariableFromObsSpace()

◆ getRecordIds()

◆ getStringVariableFromObsSpace()

◆ getValidObservationIds() [1/2]

◆ getValidObservationIds() [2/2]

◆ groupObservationsByCategoryVariable()

◆ groupObservationsByRecordNumber()

◆ isValid()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ splitObservationsIntoIndependentGroups()

◆ toAllObservations()

◆ toObservationsSplitIntoIndependentGroupsByRecordId()

◆ toObservationsSplitIntoIndependentGroupsByVariable()

◆ totalNumObservations()

◆ wereRecordsGroupedByCategoryVariable()

Member Data Documentation

◆ categoryVariable_

◆ groupBy_

◆ obsdb_

◆ obsDistribution_