{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data structures\n", "GeoST uses standardized internal data structures and data validation to ensure that the\n", "functionality that GeoST offers can always reliably be applied to parsed data. This user\n", "guide sections dives deeper into GeoST data structures. For a more basic overview of the\n", "concepts, see the [Introduction to GeoST](../getting_started/introduction.ipynb#concept).\n", "\n", "## Point and line data\n", "To describe point data (e.g. boreholes, well logs, cpts) and line data (e.g. seismics, \n", "GPR, EM) you need a minimal amount of information on the identification and position of each\n", "point/line (`Header`). For each point/line there are measurements or descriptions available \n", "of the subsurface (`Data`). The following header and data objects are used to\n", "describe point and line data:\n", "\n", "**Header objects:**\n", "* *[`PointHeader`](../api_reference/point_header.rst)* describes metadata and spatial information of point surveys.\n", "* *[`LineHeader`](../api_reference/line_header.rst)* describes metadata and spatial information of line surveys.\n", "\n", "**Data objects:**\n", "* *[`LayeredData`](../api_reference/layered_data.rst)* describes subsurface data in layers defined by tops and bottoms.\n", "* *[`DiscreteData`](../api_reference/discrete_data.rst)* describes subsurface data discretized by depth.\n", "* *LineData* NOTE: Not yet implemented\n", "\n", "These basic objects are used to build `Collections`. E.g. a [`BoreholeCollection`](../api_reference/borehole_collection.rst)\n", "is built from the combination of a [`PointHeader`](../api_reference/point_header.rst)\n", "object and a [`LayeredData`](../api_reference/layered_data.rst) objects. The below \n", "figure gives a complete overview of the object hierarchy in GeoST for point and line data.\n", "\n", "

\n", " \"GeoST\n", "

\n", "\n", "### Collection objects\n", "Collection objects are composed of an instance of Header and Data. The collection provides\n", "additional logic to maintain alignment between header and data. A collection object\n", "inherits all methods that are provided both through the child header object and the child data\n", "object. For example: you can access spatial selection methods (= header operations) as well as\n", "data slicing methods (= data operations) directly from the collection. It is recommended to\n", "work with collections by default, unless you specifically need only header or data functionality.\n", "\n", "GeoST currently offers the following collection classes:\n", "\n", "* *[`BoreholeCollection`](../api_reference/borehole_collection.rst)*: A collection of borehole data, composed of [`PointHeader`](../api_reference/point_header.rst) and [`LayeredData`](../api_reference/layered_data.rst).\n", "* *[`CptCollection`](../api_reference/cpt_collection.rst)*: A collection of cone penetration test data, composed of [`PointHeader`](../api_reference/point_header.rst) and [`DiscreteData`](../api_reference/discrete_data.rst).\n", "\n", "By default, read functions for point/line data return a collection (see: [Reading data](./reading_data.ipynb)). \n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "BoreholeCollection:\n", "# header = 67\n" ] } ], "source": [ "import geost\n", "\n", "# Load the Utrecht Science Park example borehole data\n", "boreholes_collection = geost.data.boreholes_usp()\n", "\n", "# boreholes_collection is an instance of BoreholeCollection and contains 67 boreholes\n", "print(boreholes_collection)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Header objects\n", "GeoST header objects are built on top of a [`Geopandas.Geodataframe`](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.html)\n", "to hold data, including point geometries ([`PointHeader`](../api_reference/point_header.rst))\n", "or linestring geometries ([`LineHeader`](../api_reference/line_header.rst)). The Geodataframe \n", "is the attribute named `gdf` within a header object. Each entry (row in the Geodataframe) \n", "corresponds to one point or line survey, e.g. one borehole or one seismic line. GeoST \n", "header objects offer built-in methods for changing horizontal and vertical reference \n", "systems, selecting data based on spatial conditions, export of table data and export of \n", "geometries for viewing header data in GIS.\n", "\n", "A [`PointHeader`](../api_reference/point_header.rst) requires a bare minimum of data\n", "columns to describe the data and to ensure that all built-in methods can be applied:\n", "\n", "| Column name | Validation criteria | Description |\n", "| ----------- | ------------------- | ----------- |\n", "| nr | Must be interpretable as string | Identification name/number/code of the point survey |\n", "| x | Must be of numeric type (int or float) | X-coordinate |\n", "| y | Must be of numeric type (int or float) | Y-coordinate |\n", "| surface | Must be of numeric type (int or float) and higher than end depth | Surface elevation of the point survey in m |\n", "| end | Must be of numeric type (int or float) and lower than surface elevation | End depth of the point survey in m |\n", "| geometry | Must be of type `shapely.geometry.Point` | Point geometry of the survey location |\n", "\n", "The header is not limited to just these columns. Any number of columns can be added to give\n", "additional information on surveys. Some built-in analysis methods may even add information\n", "to the header. For instance, the method [`PointHeader.get_area_labels`](../api_reference/generated/geost.base.PointHeader.get_area_labels.rst)\n", "has an argument `include_in_header` which, if set to true, adds a column with results\n", "to the header Geodataframe.\n", "\n", "If you're only interested in survey locations and/or metadata, it is adviced to directly\n", "work with the header object to avoid additional overhead caused by a parent collection \n", "object (overhead is caused by checks of the header against data after every operation to \n", "ensure header/data alignment). Read functions for point and line data (see: [Reading data](./reading_data.ipynb))\n", "return a corresponding collection object by default, but you can assign only the header to \n", "a variable in order to continue with just the header data. See the example below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EPSG:28992\n", "EPSG:5709\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nrxysurfaceendgeometry
0B31H05411395854560001.20-9.90POINT (139585 456000)
1B31H06111396004550601.20-23.00POINT (139600 455060)
2B31H07181399504552001.30-271.20POINT (139950 455200)
3B31H08031396754550872.16-4.84POINT (139675 455087)
4B31H08061396844553841.00-49.50POINT (139684 455384)
\n", "
" ], "text/plain": [ " nr x y surface end geometry\n", "0 B31H0541 139585 456000 1.20 -9.90 POINT (139585 456000)\n", "1 B31H0611 139600 455060 1.20 -23.00 POINT (139600 455060)\n", "2 B31H0718 139950 455200 1.30 -271.20 POINT (139950 455200)\n", "3 B31H0803 139675 455087 2.16 -4.84 POINT (139675 455087)\n", "4 B31H0806 139684 455384 1.00 -49.50 POINT (139684 455384)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load the Utrecht Science Park example borehole data and only assign the header data.\n", "boreholes_header = geost.data.boreholes_usp().header\n", "\n", "# Print horizontal and vertical reference system properties and the first few rows of\n", "# the boreholes header data.\n", "print(boreholes_header.horizontal_reference)\n", "print(boreholes_header.vertical_reference)\n", "boreholes_header.gdf.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data objects\n", "GeoST data objects are built on top of a [`Pandas.DataFrames`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) \n", "to store data. The actual dataframe is named `df` within a data object. Each entry (row)\n", "in the data represents a single layer (in case of [`LayeredData`](../api_reference/layered_data.rst))\n", "bounded by a top and a bottom or a single measurement (in case of [`DiscreteData`](../api_reference/discrete_data.rst))\n", "at a certain depth. One point or line survey (i.e. one row in the header) can be associated\n", "with multiple rows of data. E.g. a single borehole with 10 described layers is represented\n", "by one row in the header Geodataframe and ten rows in the data DataFrame. GeoST \n", "data objects offer built-in methods for conditional selections, slicing, basic\n", "analysis and data export.\n", "\n", "An instance of [`LayeredData`](../api_reference/layered_data.rst) requires a bare minimum of data\n", "columns to describe the data and to ensure that all built-in methods can be applied:\n", "\n", "| Column name | Validation criteria | Description |\n", "| ----------- | ------------------- | ----------- |\n", "| nr | Must be interpretable as string | Identification name/number/code of the point survey |\n", "| x | Must be of numeric type (int or float) | X-coordinate |\n", "| y | Must be of numeric type (int or float) | Y-coordinate |\n", "| x_bot | Must be of numeric type (int or float) | X-coordinate of layer bottom (only required if survey does not point straight down) |\n", "| y_bot | Must be of numeric type (int or float) | X-coordinate of layer bottom (only required if survey does not point straight down) |\n", "| surface | Must be of numeric type (int or float) and higher than end depth | Surface elevation of the point survey in m |\n", "| end | Must be of numeric type (int or float) and lower than surface elevation | End depth of the point survey in m |\n", "| top | Must be of numeric type (int or float); starts at 0; is increasing | Elevation of layer top. The first layer always starts at 0 and increases downwards |\n", "| bottom | Must be of numeric type (int or float); is larger than top; is increasing | Elevation of layer bottom |\n", "\n", "An instance of [`DiscreteData`](../api_reference/discrete_data.rst) requires a bare minimum of data\n", "columns to describe the data and to ensure that all built-in methods can be applied:\n", "\n", "| Column name | Validation criteria | Description |\n", "| ----------- | ------------------- | ----------- |\n", "| nr | Must be interpretable as string | Identification name/number/code of the point survey |\n", "| x | Must be of numeric type (int or float) | X-coordinate |\n", "| y | Must be of numeric type (int or float) | Y-coordinate |\n", "| surface | Must be of numeric type (int or float) and higher than end depth | Surface elevation of the point survey in m |\n", "| end | Must be of numeric type (int or float) and lower than surface elevation | End depth of the point survey in m |\n", "| depth | Must be of numeric type (int or float); is increasing | Depth where the measurement was taken |\n", "\n", "All other columns contain the actual data with measurements for each layer or at each depth.\n", "\n", "If you're only interested in the measurements and don't need to work with geometries or\n", "any other additional header data, it is adviced to directly work with the data object to \n", "avoid additional overhead caused by a parent collection object (overhead is caused by \n", "checks of the header against data after every operation to ensure header/data alignment). \n", "Read functions for point and line data (see: [Reading data](./reading_data.ipynb))\n", "return a corresponding collection object by default, but you can assign only the data object to \n", "a variable in order to continue with just the data. See the example below. Some\n", "read functions, such as [`read_borehole_table`](../api_reference/generated/geost.read_borehole_table.rst)\n", "provide the argument `as_collection` which defaults to True, but can be set to False to\n", "only return the [`LayeredData`](../api_reference/layered_data.rst) object in this example." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nrxysurfaceendtopbottomlithzmzmk...conscolorlutum_pctplantsshellskleibrokjesstrat_1975strat_2003strat_interdesc
0B31H05411395854560001.2-9.90.000.20KNaNNone...NoneONNaN000NoneECNaN[TEELAARDE#***#****#*] ..........................
1B31H05411395854560001.2-9.90.200.60KNaNNone...NoneBRNaN000NoneECNaN[KLEI#***#****#*] grysbruin.
2B31H05411395854560001.2-9.90.600.95VNaNNone...NoneBRNaN000NoneNINaN[VEEN#***#****#*] donkerbruin.
3B31H05411395854560001.2-9.90.952.80ZNaNZMFO...NoneGRNaN000NoneECNaN[ZAND#***#****#*] FYN TOT matig fyn# iets slib...
4B31H05411395854560001.2-9.92.804.20ZNaNZFC...NoneBRNaN000NoneBXWINaN[ZAND#***#****#*] fyn# grysbruin.
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " nr x y surface end top bottom lith zm zmk ... \\\n", "0 B31H0541 139585 456000 1.2 -9.9 0.00 0.20 K NaN None ... \n", "1 B31H0541 139585 456000 1.2 -9.9 0.20 0.60 K NaN None ... \n", "2 B31H0541 139585 456000 1.2 -9.9 0.60 0.95 V NaN None ... \n", "3 B31H0541 139585 456000 1.2 -9.9 0.95 2.80 Z NaN ZMFO ... \n", "4 B31H0541 139585 456000 1.2 -9.9 2.80 4.20 Z NaN ZFC ... \n", "\n", " cons color lutum_pct plants shells kleibrokjes strat_1975 strat_2003 \\\n", "0 None ON NaN 0 0 0 None EC \n", "1 None BR NaN 0 0 0 None EC \n", "2 None BR NaN 0 0 0 None NI \n", "3 None GR NaN 0 0 0 None EC \n", "4 None BR NaN 0 0 0 None BXWI \n", "\n", " strat_inter desc \n", "0 NaN [TEELAARDE#***#****#*] .......................... \n", "1 NaN [KLEI#***#****#*] grysbruin. \n", "2 NaN [VEEN#***#****#*] donkerbruin. \n", "3 NaN [ZAND#***#****#*] FYN TOT matig fyn# iets slib... \n", "4 NaN [ZAND#***#****#*] fyn# grysbruin. \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load the Utrecht Science Park example borehole data and only assign the data.\n", "boreholes_data = geost.data.boreholes_usp().data\n", "\n", "# Print the first few rows of boreholes data.\n", "boreholes_data.df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nrxyvertical_datumsurfacecone_penetration_test_fkcone_penetration_test_result_pkpenetration_lengthdepthelapsed_time...magnetic_inclinationmagnetic_declinationlocal_frictionpore_ratiotemperaturepore_pressure_u1pore_pressure_u2pore_pressure_u3friction_ratioend
0CPT000000009626140950.998794455358.997741NAP2.09579116908820.2NaNNaN...NoneNoneNaNNoneNoneNoneNaNNoneNaN0
1CPT000000009626140950.998794455358.997741NAP2.09579116908830.3NaNNaN...NoneNoneNaNNoneNoneNoneNaNNoneNaN0
2CPT000000009626140950.998794455358.997741NAP2.09579116908840.4NaNNaN...NoneNoneNaNNoneNoneNoneNaNNoneNaN0
3CPT000000009626140950.998794455358.997741NAP2.09579116908850.5NaNNaN...NoneNoneNaNNoneNoneNoneNaNNoneNaN0
4CPT000000009626140950.998794455358.997741NAP2.09579116908860.6NaNNaN...NoneNoneNaNNoneNoneNoneNaNNoneNaN0
\n", "

5 rows × 33 columns

\n", "
" ], "text/plain": [ " nr x y vertical_datum surface \\\n", "0 CPT000000009626 140950.998794 455358.997741 NAP 2.0 \n", "1 CPT000000009626 140950.998794 455358.997741 NAP 2.0 \n", "2 CPT000000009626 140950.998794 455358.997741 NAP 2.0 \n", "3 CPT000000009626 140950.998794 455358.997741 NAP 2.0 \n", "4 CPT000000009626 140950.998794 455358.997741 NAP 2.0 \n", "\n", " cone_penetration_test_fk cone_penetration_test_result_pk \\\n", "0 9579 11690882 \n", "1 9579 11690883 \n", "2 9579 11690884 \n", "3 9579 11690885 \n", "4 9579 11690886 \n", "\n", " penetration_length depth elapsed_time ... magnetic_inclination \\\n", "0 0.2 NaN NaN ... None \n", "1 0.3 NaN NaN ... None \n", "2 0.4 NaN NaN ... None \n", "3 0.5 NaN NaN ... None \n", "4 0.6 NaN NaN ... None \n", "\n", " magnetic_declination local_friction pore_ratio temperature pore_pressure_u1 \\\n", "0 None NaN None None None \n", "1 None NaN None None None \n", "2 None NaN None None None \n", "3 None NaN None None None \n", "4 None NaN None None None \n", "\n", " pore_pressure_u2 pore_pressure_u3 friction_ratio end \n", "0 NaN None NaN 0 \n", "1 NaN None NaN 0 \n", "2 NaN None NaN 0 \n", "3 NaN None NaN 0 \n", "4 NaN None NaN 0 \n", "\n", "[5 rows x 33 columns]" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Load the Utrecht Science Park example CPT data and only assign the data.\n", "cpt_data = geost.data.cpts_usp().data\n", "\n", "# Print the first few rows of CPT data.\n", "cpt_data.df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Model data\n", "GeoST supports working with model data and offers methods to combine these data with\n", "point and line data. Model data does not follow the same header/data approach as point\n", "and line data. Instead there are generic model classes, of which some have an\n", "implementation that adds specific functionality for that model. An example of this is\n", "the [`VoxelModel`](../api_reference/voxelmodel.rst) as a generic model class and [`GeoTOP`](../api_reference/bro_geotop.rst)\n", "being a specific implementation of a voxel model. GeoST currently support the following \n", "generic models and implementations:\n", "\n", "**Generic models and implementations**\n", "* *[`VoxelModel`](../api_reference/voxelmodel.rst)*: Class for voxel models, with data \n", "stored in the `ds` attribute, an [`Xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html).\n", " * Implementations: [`GeoTOP`](../api_reference/bro_geotop.rst)\n", "* *`LayerModel`*: Class for layer models, not yet implemented\n", " * Implementations: None\n", "\n", "

\n", " \"GeoST\n", "

\n", "\n", "### Voxel models\n", "The [`VoxelModel`](../api_reference/voxelmodel.rst) class stores data in the `ds` \n", "attribute, which is an [`Xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html).\n", "A custom voxel model can be instantiated from a NetCDF file. For this, see the documentation of the \n", "[`VoxelModel.from_netcdf`](../api_reference/generated/geost.models.VoxelModel.from_netcdf.rst) class constructor.\n", "An instance of [`VoxelModel`](../api_reference/voxelmodel.rst) offers basic methods for \n", "selecting, slicing and exporting models.\n", "\n", "For more guidance on using a Voxel model within GeoST, see the [BRO GeoTOP](../user_guide/bro_geotop.ipynb)\n", "section in the user guide.\n", "\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "default", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.7" } }, "nbformat": 4, "nbformat_minor": 2 }