Data structures#

GeoST uses standardized internal data structures and data validation to ensure that the functionality it offers can always be applied reliably. Before we discuss the relevant data structures, we introduce some terminology. Subsurface data come in many different forms (e.g., boreholes, CPTs, or 3D models), but we distinguish between two main types:

Survey: the actual measurements (i.e., raw data) of the subsurface. These can include boreholes, CPTs, well logs, seismic or EM lines, and others.
Model: for example, a 3D voxel or layer model or a geological map. These are the result of analyses using survey data (e.g., Kriging interpolation) and are therefore considered an interpretation of the subsurface.

GeoST provides different data structures to handle these different kinds of data. These will be introduced in this section of the user guide.

Collection#

For survey data, the core data structure is a geost.Collection, which holds the data in a header and data table. Basically, these tables contain:

header: metadata and spatial information for each survey.
data: the logged data of the surveys.

The header and data tables have a one-to-many relationship: one survey (e.g., a borehole) is represented by one row in the header and multiple rows in the data.

Typically, available types of subsurface data include point-like data, such as boreholes, CPTs, and well logs, or line-like data, such as seismic, GPR, and EM data. These can all be stored in a Collection. By default, the available reader functions for different types of survey data return a Collection (see Survey data section in this user guide).

While working with a Collection, selections change the contents of the header and data tables, but the Collection automatically maintains alignment between the two. Therefore, users can safely select and analyse the data while ensuring consistency. It is recommended to work with a Collection by default, unless you specifically need to work with only the header or data table.

Let’s first look at a Collection containing borehole data:

import geost

# Load the Utrecht Science Park example borehole data
collection = geost.data.boreholes_usp()
collection

Collection
  header (rows, columns) : (67, 6)
  data (rows, columns)   : (1398, 32)
crs: RD
vertical datum: NAP height

Overview of attributes and methods#

The Collection class implements many different attributes as well as analysis and selection methods. A short summary of the relevant attributes and methods is presented here. A full overview can be found in the Collection API reference.

Attributes

data: Data table of the Collection.
header: Header table of the Collection.
crs: Current coordinate reference system (CRS) of the Collection.
vertical_datum: Current vertical datum of the Collection.

Reference systems

set_crs: Set the CRS of the Collection.
to_crs: Convert the current Collection CRS to the specified CRS.
set_vertical_datum: Similar to set_crs, but for the vertical datum (e.g., NAP = EPSG:5709).

Analysis

get_cumulative_thickness: Return the cumulative thickness for any specified criteria.
get_layer_top: Return the top depth at which a specified layer occurs.

Spatial selections

select_within_bbox: Select data points in the Collection within a bounding box.
select_with_points: Select data points in the Collection within a specified distance of other point geometries.
select_with_lines: Select data points in the Collection within a specified distance of line geometries.
select_within_polygons: Select data points in the Collection within polygon geometries.

Conditional selections

select_by_values: Select data points in the Collection based on the presence of certain values in one or more data columns.
select_by_length: Select data points in the Collection based on length requirements.
select_by_depth: Select data points in the Collection based on depth constraints.

Slicing

slice_depth_interval: Slice boreholes in the Collection to the specified depth interval.
slice_by_values: Slice boreholes in the Collection based on values (e.g., keep only sand layers and remove all others).

Header table#

In a Collection, the header table is always a geopandas.GeoDataFrame instance, which the Collection ensures internally. The header contains the spatial location of each survey in the "geometry" column, as well as metadata such as the surface level, end depth, and other information. Let’s take a look at the header:

print(f"Type of header: {type(collection.header)}")
collection.header.head()

Type of header: <class 'geopandas.geodataframe.GeoDataFrame'>

	nr	x	y	surface	end	geometry
0	B31H0541	139585	456000	1.20	-9.90	POINT (139585 456000)
1	B31H0611	139600	455060	1.20	-23.00	POINT (139600 455060)
2	B31H0718	139950	455200	1.30	-271.20	POINT (139950 455200)
3	B31H0803	139675	455087	2.16	-4.84	POINT (139675 455087)
4	B31H0806	139684	455384	1.00	-49.50	POINT (139684 455384)

In the Collection above, the geometry column contains shapely.Point geometries. For 2D line data (e.g., seismic data), this would instead contain shapely.LineString geometries. Each entry in the header (i.e., row in the GeoDataFrame) corresponds to a specific survey, such as a borehole or seismic line.

The header is not limited to the columns shown above. Any number of columns can be added to the header to provide additional information about the surveys. Various analysis methods can be used for this; see, for example, the Collection.spatial_join method.

If you are only interested in survey locations and/or metadata, it is advisable to work directly with the header object to avoid the (small) additional overhead of a Collection object. This overhead is caused by checks of the header against the data after operations to ensure alignment between the two. If you are only working with the header or data table, GeoST functionality is still available through DataFrame and GeoDataFrame accessors, which we will cover in a later section.

Data table#

The data table is generally a pandas.DataFrame instance, although it can also be a GeoDataFrame. The data table holds all the logged data for each survey. Let’s take a look at it:

print(f"Type of data: {type(collection.data)}")
collection.data

Type of data: <class 'pandas.DataFrame'>

	nr	x	y	surface	end	top	bottom	lith	zm	zmk	...	cons	color	lutum_pct	plants	shells	kleibrokjes	strat_1975	strat_2003	strat_inter	desc
0	B31H0541	139585	456000	1.200	-9.900	0.00	0.20	K	NaN	NaN	...	NaN	ON	NaN	0	0	0	NaN	EC	NaN	[TEELAARDE#*#*#] ..........................
1	B31H0541	139585	456000	1.200	-9.900	0.20	0.60	K	NaN	NaN	...	NaN	BR	NaN	0	0	0	NaN	EC	NaN	[KLEI#*#*#] grysbruin.
2	B31H0541	139585	456000	1.200	-9.900	0.60	0.95	V	NaN	NaN	...	NaN	BR	NaN	0	0	0	NaN	NI	NaN	[VEEN#*#*#] donkerbruin.
3	B31H0541	139585	456000	1.200	-9.900	0.95	2.80	Z	NaN	ZMFO	...	NaN	GR	NaN	0	0	0	NaN	EC	NaN	[ZAND#*#*#] FYN TOT matig fyn# iets slib...
4	B31H0541	139585	456000	1.200	-9.900	2.80	4.20	Z	NaN	ZFC	...	NaN	BR	NaN	0	0	0	NaN	BXWI	NaN	[ZAND#*#*#] fyn# grysbruin.
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
1393	B32C1893	141266	455989	2.328	-2.172	0.00	0.50	Z	NaN	ZZF	...	NaN	BR	NaN	0	0	0	NaN	NaN	NaN	BRON:GEF-BESTAND;0.00;0.50;Zs3h2 PU2;ZZF;BR;
1394	B32C1893	141266	455989	2.328	-2.172	0.50	1.10	Z	NaN	ZZF	...	NaN	GR	NaN	0	0	0	NaN	NaN	NaN	BRON:GEF-BESTAND;0.50;1.10;Zs3 KLE2;ZZF;GR DO;
1395	B32C1893	141266	455989	2.328	-2.172	1.10	1.40	K	NaN	NaN	...	CMST	GR	NaN	0	0	0	NaN	NaN	NaN	BRON:GEF-BESTAND;1.10;1.40;Ks2h1;;GR TBR;KMST
1396	B32C1893	141266	455989	2.328	-2.172	1.40	2.10	Z	NaN	ZZF	...	NaN	GR	NaN	0	0	0	NaN	NaN	NaN	BRON:GEF-BESTAND;1.40;2.10;Zs2 SLI1;ZZF;GR DO;
1397	B32C1893	141266	455989	2.328	-2.172	2.10	4.50	Z	NaN	ZZF	...	NaN	GR	NaN	0	0	0	NaN	NaN	NaN	BRON:GEF-BESTAND;2.10;4.50;Zs2;ZZF;GR;

1398 rows × 32 columns

Here we see that a single survey contains multiple rows, which in this case represent “layers”, as we are looking at borehole data. For survey data, we can generally distinguish between “layered” and “discrete” data:

Layered data contains information that is logged in terms of layers (i.e., depth intervals over which properties are the same), with “top” and “bottom” information for each layer.
Discrete data contains information that is logged at discrete intervals (e.g., every 20 cm), with “depth” information for each depth interval.

GeoST can treat both types of data interchangeably, and provides ways to combine one type with the other. As with the header table, the data table is not limited to the columns shown above, and any number of columns can be added during analysis.

If you are only interested in the survey measurements and do not need to work with geometries or any other additional header data, it is advisable to work directly with the data table to avoid the overhead of a Collection (i.e., maintaining alignment between the header and data tables). See the accessor section. Several reader functions provide the option to return a DataFrame instead of a Collection by setting as_collection=False when using the function.

Positional columns#

GeoST requires certain data columns to be present to ensure that Collection methods can work. These are referred to as “positional columns”. The required columns differ depending on the method being used. For example, Collection.slice_depth_interval requires depth information about the surface level of surveys and the depth of layers, while Collection.select_with_points requires valid geometries.

The presence of depth information or a valid geometry is therefore required for these methods to work. However, both are optional at the Collection level. The slice_depth_interval method does not need geometry to work, while select_with_points does not need depth information. Their presence is therefore only required when you want to use the corresponding methods.

This was chosen as a design principle to provide maximum flexibility for users with different needs. The only mandatory requirement is a column that identifies each individual survey (e.g., "nr") in both the header and data tables.

Note

See the Positional columns section on the Survey data page of this User guide for specific information on positional columns.

GeoST accessor#

When you only need to work with either the header or data table, all methods available in Collection are also available when working directly with the header GeoDataFrame or data DataFrame. This is achieved through a so-called “accessor”, which extends the functionality of DataFrames. Under the hood, every Collection method also uses this accessor.

Below, we briefly demonstrate how this works by comparing the usage of Collection methods with the corresponding accessor methods. The accessor can be accessed by calling .gst on a GeoDataFrame or DataFrame.

# Create separate `header` and `data` variables for the demonstration.
header = collection.header
data = collection.data

print(header.gst)
print(data.gst)

<geost.accessor.GeostFrame object at 0x7fe7f4c19310>
<geost.accessor.GeostFrame object at 0x7fe7f8cdf5b0>

As you can see, for both the header and data table, it prints geost.accessor.GeostFrame objects. This is the object where most GeoST methods are actually implemented, which is why any method available in a Collection is also accessible through the .gst accessor.

Let’s first compare the usage of select_within_bbox, a method that typically operates on the header table because it is a spatial method and the header contains the spatial information about the surveys. As the name suggests, this method selects the surveys located within a specific bounding box extent. All we need to do to call this method on the header GeoDataFrame is use .gst in between, as shown below.

collection_select = collection.select_within_bbox(139_500, 455_000, 140_000, 455_500)
header_select = header.gst.select_within_bbox(139_500, 455_000, 140_000, 455_500)

print("collection_select.data:", collection_select.header, sep="\n")
print("header_select:", header_select, sep="\n")

header_select.gst  # Selection result also has the .gst accessor and methods available

collection_select.data:
          nr       x       y  surface      end               geometry
 B31H0611  139600  455060    1.200  -23.000  POINT (139600 455060)
 B31H0718  139950  455200    1.300 -271.200  POINT (139950 455200)
 B31H0803  139675  455087    2.160   -4.840  POINT (139675 455087)
 B31H0806  139684  455384    1.000  -49.500  POINT (139684 455384)
 B31H0807  139684  455405    1.000  -49.500  POINT (139684 455405)
 B31H0810  139901  455401    1.000  -50.000  POINT (139901 455401)
B31H1694  139520  455125    1.200   -0.800  POINT (139520 455125)
B31H1695  139660  455290    1.100   -0.400  POINT (139660 455290)
B31H1696  139820  455020    1.300   -3.700  POINT (139820 455020)
B31H3284  139621  455080    2.010   -2.990  POINT (139621 455080)
B31H3285  139521  455181    2.157   -2.343  POINT (139521 455181)
B31H3286  139636  455194    1.492   -3.008  POINT (139636 455194)
B31H3287  139647  455396    1.430   -6.070  POINT (139647 455396)
B31H3288  139534  455397    1.560   -6.440  POINT (139534 455397)
header_select:
          nr       x       y  surface      end               geometry
 B31H0611  139600  455060    1.200  -23.000  POINT (139600 455060)
 B31H0718  139950  455200    1.300 -271.200  POINT (139950 455200)
 B31H0803  139675  455087    2.160   -4.840  POINT (139675 455087)
 B31H0806  139684  455384    1.000  -49.500  POINT (139684 455384)
 B31H0807  139684  455405    1.000  -49.500  POINT (139684 455405)
 B31H0810  139901  455401    1.000  -50.000  POINT (139901 455401)
B31H1694  139520  455125    1.200   -0.800  POINT (139520 455125)
B31H1695  139660  455290    1.100   -0.400  POINT (139660 455290)
B31H1696  139820  455020    1.300   -3.700  POINT (139820 455020)
B31H3284  139621  455080    2.010   -2.990  POINT (139621 455080)
B31H3285  139521  455181    2.157   -2.343  POINT (139521 455181)
B31H3286  139636  455194    1.492   -3.008  POINT (139636 455194)
B31H3287  139647  455396    1.430   -6.070  POINT (139647 455396)
B31H3288  139534  455397    1.560   -6.440  POINT (139534 455397)

<geost.accessor.GeostFrame at 0x7fe7f8c967b0>

We can see that the selection result is exactly the same. After selecting from the header table, the .gst accessor remains available on the selection result, allowing us to make further selections or chain several selection methods, for example.

Methods that would typically operate on the data table, such as slice_by_values, work in exactly the same way. All we need to do to call this method on the "data" DataFrame is use .gst in between, as shown below.

# Select boreholes which contain clay anywhere as the main lithology.
collection_select = collection.slice_by_values("lith", "K")
data_select = data.gst.slice_by_values("lith", "K")

print("collection_select.data:", collection_select.data[["nr", "lith"]], sep="\n")
print("data_select:", data_select[["nr", "lith"]], sep="\n")

data_select.gst  # Selection result also has the .gst accessor and methods available

collection_select.data:
            nr lith
   B31H0541    K
   B31H0541    K
   B31H0611    K
   B31H0611    K
  B31H0718    K
...        ...  ...
B32C1881    K
B32C1889    K
B32C1891    K
B32C1891    K
B32C1893    K

[261 rows x 2 columns]
data_select:
            nr lith
   B31H0541    K
   B31H0541    K
   B31H0611    K
   B31H0611    K
  B31H0718    K
...        ...  ...
B32C1881    K
B32C1889    K
B32C1891    K
B32C1891    K
B32C1893    K

[261 rows x 2 columns]

<geost.accessor.GeostFrame at 0x7fe7f4c14830>

Again the selection result is exactly the same and after the selection, the .gst accessor remains available.

Model data#

GeoST also supports working with (geological) model data and provides methods for combining these with survey data. See the Model data section in the user guide for more detailed information about working with model data in GeoST.