{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to GeoST\n", "\n", "This introduction will cover some of the key concepts and basic features of `GeoST` to help you get started. `GeoST` depends on popular data science libraries [Pandas](https://pandas.pydata.org/docs/index.html) and [GeoPandas](https://geopandas.org/en/stable/index.html) but `GeoST` provides readily available, frequently used selections on data held in [`pandas.DataFrame`](https://pandas.pydata.org/docs/reference/frame.html) or [`geopandas.GeoDataFrame`](https://geopandas.org/en/stable/docs/reference/geodataframe.html) objects. This makes GeoST an easy to use option for less experienced Python users while more experienced users can easily access the underlying DataFrames and apply their own analyses.\n", "\n", "GeoST is designed to work with many different kinds of subsurface data that are available. The package is a work-in-progress and we aim to support an increasing number of data sources. Below is a list of different data sources which are currently supported or will be supported by GeoST in the future:\n", "\n", "**From local files**:\n", "- Tabular data of borehole, CPT, etc. (.parquet, .csv)\n", "- Geological boreholes xml (BHR-G)\n", "- Geotechnical boreholes xml (BHR-GT)\n", "- Pedological boreholes xml (BHR-P)\n", "- Cone Penetration Test xml/gef (CPT)\n", "- Pedological soilprofile descriptions xml (SFR)\n", "- BORIS xml (TNO borehole description software)\n", "- NLOG boreholes Excel\n", "\n", "**Directly from the [BRO REST-API](https://www.bro-productomgeving.nl/bpo/latest/url-s-publieke-rest-services)**:\n", "- BHR-G\n", "- BHR-GT\n", "- BHR-P\n", "- CPT\n", "- SFR\n", "\n", "**BRO models**:\n", "- GeoTOP: from local NetCDF or directly via [OPeNDAP server](https://dinodata.nl/opendap/)\n", "\n", "*Planned*:\n", "- BRO/PDOK geopackages: [BHR-G](https://service.pdok.nl/bzk/bro-geologisch-booronderzoek/atom/index.xml), [BHR-GT](https://service.pdok.nl/bzk/bro-geotechnischbooronderzoek/atom/v1_0/index.xml), [BHR-P](https://service.pdok.nl/bzk/brobhrpvolledigeset/atom/v1_1/index.xml), [CPT](https://service.pdok.nl/bzk/brocptvolledigeset/atom/v1_0/index.xml), [SFR](https://service.pdok.nl/bzk/bodem/bro-wandonderzoek/atom/index.xml)\n", "- Well logs LAS/ASCII\n", "- REGIS II\n", "- Dino xml geological boreholes \n", "- BHR-G gef \n", "- Soilmap of the Netherlands\n", "\n", "GeoST also plans support for several Geophysical data sources such as Seismic, ERT, EM and others.\n", "\n", "## Concept\n", "The core data structure is a [`geost.Collection`](../api_reference/collection.rst). This holds all the spatial information of any kind of data source in a **\"header\"** attribute (i.e. header table), and the corresponding data in a **\"data\"** attribute (i.e. data table). So for example, a set of 100 boreholes is held in a `Collection` where the **\"header\"** contains one row per data entry and provides information about the id, location, surface level and depths and the **\"data\"** has the survey information for each described layer. The structure of a `Collection` is illustrated below.\n", "\n", "

\n", " \"collection\"\n", "

\n", "\n", "When working with a Collection, GeoST automatically keeps track of the alignment and thus makes sure each data entry occurs in both the **\"header\"** and **\"data\"** attributes. For example, when a user deletes an individual borehole entry from the **\"header\"**, the Collection ensures it is deleted from the **\"data\"** as well.\n", "\n", "\n", "```{note}\n", "**User guide:**\n", "Checkout the [Data structures](../user_guide/data_structures.ipynb) section in the user guide for a more detailed explanation of the GeoST data structures.\n", "```\n", "\n", "## The Basics\n", "Data is usually loaded through various reader functions (see [API reference](../api_reference/io.rst)). For this tutorial, `GeoST` provides a test set of boreholes in the area of the Utrecht Science Park which can be directly loaded as a `Collection`. Let's read the data, print the result to see what it says and also plot the locations to get an idea where we are:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collection:\n", " header (rows, columns): (67, 5)\n", " data (rows, columns): (1398, 32)\n", "crs: EPSG:28992\n", "vertical datum: EPSG:5709\n", "# surveys = 67\n", "\n" ] }, { "data": { "text/html": [ "
Make this Notebook Trusted to load map: File -> Trust Notebook
" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import geost\n", "\n", "usp_boreholes = geost.data.boreholes_usp()\n", "print(usp_boreholes)\n", "usp_boreholes.header.explore() # Interactive plot of the borehole locations." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, 'usp_boreholes' is of the type [Collection](../api_reference/borehole_collection.rst). Additionally, it shows the number of rows and columns in the header and data tables, the coordinate reference system and the number of surveys.\n", "\n", "The \"header\" attribute in a `Collection` contains all the information about each borehole such as the ID, x- and y-coordinates and further metadata. Also it contains geometry objects for each borehole which allows for spatial selections and exports to GIS-supported formats etc. that are provided by `GeoST`. The header attribute is a [`geopandas.GeoDataFrame`](https://geopandas.org/en/stable/docs/reference/geodataframe.html) instance. Let's see what the attribute looks like by printing it:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nrxysurfacegeometry
0B31H05411395854560001.20POINT (139585 456000)
1B31H06111396004550601.20POINT (139600 455060)
2B31H07181399504552001.30POINT (139950 455200)
3B31H08031396754550872.16POINT (139675 455087)
4B31H08061396844553841.00POINT (139684 455384)
\n", "
" ], "text/plain": [ " nr x y surface geometry\n", "0 B31H0541 139585 456000 1.20 POINT (139585 456000)\n", "1 B31H0611 139600 455060 1.20 POINT (139600 455060)\n", "2 B31H0718 139950 455200 1.30 POINT (139950 455200)\n", "3 B31H0803 139675 455087 2.16 POINT (139675 455087)\n", "4 B31H0806 139684 455384 1.00 POINT (139684 455384)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "usp_boreholes.header.head() # First five rows of the header data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the header is a `GeoDataFrame`, we have direct access to all methods provided by GeoDataFrames. Therefore, the above interactive plot of the borehole locations was easily created using the `.explore()` method. More experienced Python users can therefore use the header to do any customized operation with GeoDataFrames they would normally do. \n", "\n", "The other key attribute of a collection is the \"data\" attribute which is a [`pandas.DataFrame`](https://pandas.pydata.org/docs/reference/frame.html) instance. This contains the actual logged data (i.e. layer descriptions) of the boreholes. In this case, the \"data\" attribute contains \"layered\" data because the borehole data is logged in terms of layers (i.e. depth intervals over which properties are the same) with \"top\" and \"bottom\". Let's see what it looks like:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nrxysurfaceendtopbottomlithzmzmk...conscolorlutum_pctplantsshellskleibrokjesstrat_1975strat_2003strat_interdesc
0B31H05411395854560001.2-9.90.000.20KNaNNaN...NaNONNaN000NaNECNaN[TEELAARDE#***#****#*] ..........................
1B31H05411395854560001.2-9.90.200.60KNaNNaN...NaNBRNaN000NaNECNaN[KLEI#***#****#*] grysbruin.
2B31H05411395854560001.2-9.90.600.95VNaNNaN...NaNBRNaN000NaNNINaN[VEEN#***#****#*] donkerbruin.
3B31H05411395854560001.2-9.90.952.80ZNaNZMFO...NaNGRNaN000NaNECNaN[ZAND#***#****#*] FYN TOT matig fyn# iets slib...
4B31H05411395854560001.2-9.92.804.20ZNaNZFC...NaNBRNaN000NaNBXWINaN[ZAND#***#****#*] fyn# grysbruin.
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ " nr x y surface end top bottom lith zm zmk ... \\\n", "0 B31H0541 139585 456000 1.2 -9.9 0.00 0.20 K NaN NaN ... \n", "1 B31H0541 139585 456000 1.2 -9.9 0.20 0.60 K NaN NaN ... \n", "2 B31H0541 139585 456000 1.2 -9.9 0.60 0.95 V NaN NaN ... \n", "3 B31H0541 139585 456000 1.2 -9.9 0.95 2.80 Z NaN ZMFO ... \n", "4 B31H0541 139585 456000 1.2 -9.9 2.80 4.20 Z NaN ZFC ... \n", "\n", " cons color lutum_pct plants shells kleibrokjes strat_1975 strat_2003 \\\n", "0 NaN ON NaN 0 0 0 NaN EC \n", "1 NaN BR NaN 0 0 0 NaN EC \n", "2 NaN BR NaN 0 0 0 NaN NI \n", "3 NaN GR NaN 0 0 0 NaN EC \n", "4 NaN BR NaN 0 0 0 NaN BXWI \n", "\n", " strat_inter desc \n", "0 NaN [TEELAARDE#***#****#*] .......................... \n", "1 NaN [KLEI#***#****#*] grysbruin. \n", "2 NaN [VEEN#***#****#*] donkerbruin. \n", "3 NaN [ZAND#***#****#*] FYN TOT matig fyn# iets slib... \n", "4 NaN [ZAND#***#****#*] fyn# grysbruin. \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "usp_boreholes.data.head() # First five rows of the data table." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Also with the \"data\" attribute, we have direct access to all methods provided by DataFrames and more experienced Python users can use it to do any customized operation. The \"data\" attribute of this collection contains 32 different columns that hold the relevant borehole data and describes characteristics such as lithology, sand grain size, plant remains and others.\n", "\n", "### Positional reference\n", "A collection contains all spatial information about the data, both horizontally and vertically. These attributes can be accessed through the \"crs\" (coordinate reference system) and \"vertical_datum\" attributes:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EPSG:28992\n", "EPSG:5709\n" ] } ], "source": [ "print(usp_boreholes.crs)\n", "print(usp_boreholes.vertical_datum)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These attributes can be used to reproject the data. For example, changing the Dutch \"Rijksdriehoekstelsel\" coordinates to \"WGS84\" coordinates. Any reprojection automatically updates the coordinates in the data. Let's change the coordinate reference system in \"usp_boreholes\" and checkout the \"header\" again to see this:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " nr x y surface geometry\n", "0 B31H0541 5.162268 52.092042 1.20 POINT (5.16227 52.09204)\n", "1 B31H0611 5.162530 52.083594 1.20 POINT (5.16253 52.08359)\n", "2 B31H0718 5.167630 52.084862 1.30 POINT (5.16763 52.08486)\n", "3 B31H0803 5.163622 52.083839 2.16 POINT (5.16362 52.08384)\n", "4 B31H0806 5.163740 52.086508 1.00 POINT (5.16374 52.08651)\n", "EPSG:4326\n" ] } ], "source": [ "usp_boreholes.to_crs(4326) # Change from RD to WGS 84\n", "print(usp_boreholes.header.head(), usp_boreholes.crs, sep=\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the coordinates in the \"x\" and \"y\" columns have indeed been changed to latitude, longitude coordinates.\n", "\n", "### Selections and slices\n", "There are several ways to make subsets from a Collection, such as:\n", "\n", "**Spatial selections**\n", "- `select_within_bbox` - Select data points in the Collection within a bounding box\n", "- `select_with_points` - Select data points in the Collection within distance to other point geometries\n", "- `select_with_lines` - Select data points in the Collection within distance from line geometries\n", "- `select_within_polygons` - Select data points in the Collection within polygon geometries\n", "\n", "**Conditional selections**\n", "- `select_by_values` - Select data points in the Collection based on the presence of certain values in one or more of the data columns\n", "- `select_by_length` - Select data points in the Collection based on length requirements \n", "- `select_by_depth` - Select data points in the Collection based on depth constraints\n", "\n", "**Slicing**\n", "- `slice_depth_interval` - Slice boreholes in the Collection down to the specified depth interval\n", "- `slice_by_values` - Slice boreholes in the Collection based on value (e.g. only sand layers, remove others).\n", "\n", "We will not go through each of these methods in this quick start but please see the [API Reference](../api_reference.md) for more details." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.3" } }, "nbformat": 4, "nbformat_minor": 2 }