Reading XML files#
This example demonstrates how GeoST extracts data from XML files (such as BRO XML files) and how to retrieve additional data that is not extracted by default by extending the methodology.
GeoST reads data from XML files using a schema that describes the structure of the file
(i.e., where to find specific elements) and determines where the information ends up in the
resulting Collection. Under the hood, GeoST uses the
lxml library for XML parsing, which we will also use in this
tutorial to demonstrate some basic principles.
Borehole and CPT data in the Netherlands are distributed through a public national database: the BRO (“Basis Registratie Ondergrond”). The BRO contains five main sources of subsurface data:
Geological boreholes (BHR-G objects)
Geotechnical boreholes (BHR-GT objects or BHR-GT-samples when retrieving grain size sample data)
Pedological boreholes (BHR-P objects)
Cone Penetration Tests (CPT objects)
Pedological soil profile descriptions (SFR objects)
Downloading any of the above data sources from BROloket results in a set of XML files for the requested objects. GeoST can be used to read the XML files for each of these data sources (see the available read functions for each object).
Sometimes, XML data for one of the above objects may also be provided by companies (e.g., Wiertsema). These files can differ slightly in structure from BRO XML files, but GeoST can work with these files as well. For this tutorial, we will demonstrate how data is extracted from a BHR-G XML file downloaded from the BRO. The principle is the same for the other data sources.
We will begin with the necessary imports.
from lxml import etree
import geost
from geost.io import xml
bhrg_file = geost.data.bhrg_bro() # BHR-G XML file for tutorial
Downloading file 'bhrg_bro.xml' from 'https://github.com/Deltares-research/geost/raw/main/data/bhrg_bro.xml' to '/home/runner/.cache/geost'.
Reading with default schema#
As mentioned above, a schema is used to read XML files, and GeoST contains predefined schemas
for each supported data source. These schemas are used to retrieve the most important header
attributes (e.g., "nr", "x", "y", and "surface") as well as basic data attributes.
For BHR-G objects, these include attributes such as "top", "bottom", and "lithology".
If you are only interested in basic attributes or want to perform a quick inspection, you can
simply use the default reader functions without worrying about schemas or other details. The
code below shows how easily a BHR-G file can be read. The result is a Collection
containing a header and data attribute.
bhrg = geost.read_bhrg(bhrg_file)
print(bhrg.header)
print(bhrg.data)
print(f"Retrieved header attributes: {bhrg.header.columns}")
print(f"Retrieved data attributes: {bhrg.data.columns}")
nr surface end x y \
0 BHR000000396406 0.69 -2.31 126148.999929 452161.999985
geometry
0 POINT (126149 452162)
nr x y surface end top bottom \
0 BHR000000396406 126148.999929 452161.999985 0.69 -2.31 0.00 0.25
1 BHR000000396406 126148.999929 452161.999985 0.69 -2.31 0.25 1.60
2 BHR000000396406 126148.999929 452161.999985 0.69 -2.31 1.60 2.00
3 BHR000000396406 126148.999929 452161.999985 0.69 -2.31 2.00 2.50
4 BHR000000396406 126148.999929 452161.999985 0.69 -2.31 2.50 3.00
soilNameNEN5104
0 zwakZandigeKlei
1 sterkSiltigeKlei
2 zwakZandigeKlei
3 sterkZandigeKlei
4 zwakSiltigZand
Retrieved header attributes: Index(['nr', 'surface', 'end', 'x', 'y', 'geometry'], dtype='str')
Retrieved data attributes: Index(['nr', 'x', 'y', 'surface', 'end', 'top', 'bottom', 'soilNameNEN5104'], dtype='str')
Schema#
We can see that using the predefined schema already retrieves many of the important
attributes from the XML file. However, some users may need additional data attributes because
they are interested in other information that may be relevant to their specific questions. For
this, we will explore the schema and the XML structure in more detail.
Let’s first take a look at the predefined schema that is used by default by
geost.read_bhrg. We will use pprint to display the structure of the schema more clearly.
from pprint import pprint
bhrg_schema = xml.schemas.bhrg[
"BRO"
] # This would be `xml.schemas.bhrgt` for BHR-GT files
pprint(
bhrg_schema, sort_dicts=False
) # Use `sort_dicts=False` to keep the order of the schema
{'payload_root': 'dispatchDocument',
'nr': {'xpath': 'brocom:broId'},
'location': {'xpath': 'standardizedLocation/brocom:location/gml:pos',
'resolver': <function parse_coordinates at 0x7d2d4e756ae0>,
'el-attr': 'text'},
'crs': {'xpath': 'standardizedLocation/brocom:location',
'resolver': <function parse_crs at 0x7d2d4e756820>},
'surface': {'xpath': 'deliveredVerticalPosition/bhrgcom:offset',
'resolver': <function safe_float at 0x7d2d4e756c40>,
'el-attr': 'text'},
'vertical_datum': {'xpath': 'deliveredVerticalPosition/bhrgcom:verticalDatum',
'resolver': <function parse_vertical_datum at 0x7d2d4e756980>,
'el-attr': 'text'},
'end': {'xpath': 'boring/bhrgcom:Boring/bhrgcom:finalDepthBoring',
'resolver': <function safe_float at 0x7d2d4e756c40>,
'el-attr': 'text'},
'data': {'xpath': 'boreholeSampleDescription/bhrgcom:BoreholeSampleDescription/bhrgcom:descriptiveBoreholeLog/bhrgcom:DescriptiveBoreholeLog',
'resolver': <function process_bhrgt_data at 0x7d2d4e7571c0>,
'layer-attributes': ['upperBoundary',
'lowerBoundary',
'soilNameNEN5104']}}
You may already recognize some of the header attributes from the Collection above,
such as "nr", "crs", and "surface". The simplest way to explain the schema above in
relation to the resulting Collection is that the "data" key determines what is
included in the data table, while all other keys end up in the header table. Each key in the
schema contains a subdictionary that “tells geost.read_bhrg what to do” for that key.
The dictionaries for keys that end up in the header can contain the keys "xpath",
"resolver", and "el-attr". As you can see, they can contain one or all of these keys.
Below is a short description of each:
"xpath": Path to the desired element in the XML structure. This is always required; otherwise, nothing will be retrieved."resolver": Python function (if necessary) to extract or transform data in a specific way. For example, it can be used to convert a text value to a float."el-attr": XML elements can have attributes, such as a"text"attribute. This ensures that the specified attribute is used as the result or as input for the resolver.
The dictionary under the "data" key in the schema also contains "xpath" and "resolver",
but "el-attr" cannot be used here. Instead, it contains a "layer-attributes" key which,
as the name suggests, extracts the desired attributes for each borehole layer in the BHR-G
object. For now, the predefined schema only extracts the "upperBoundary",
"lowerBoundary", and "soilNameNEN5104" attributes of each layer. We will show how the
schema above can be extended to retrieve additional data.
Retrieving additional attributes#
Since "upperBoundary", "lowerBoundary", and "soilNameNEN5104" may not provide enough
information for every purpose, it may be desirable to retrieve additional layer attributes.
Let’s take a look at an element in the XML file that contains the data for a single layer and
see what additional attributes are available. Normally, you could inspect the entire XML file
in a text editor such as Notepad++. In this tutorial, however, we will retrieve and print
the layer element using lxml to see which attributes are present.
xml_tree = etree.parse(
bhrg_file
).getroot() # Read the XML file and get the root element
# Unfortunately, the path to a layer element from the root is really long
path_layer_element = "dispatchDocument/BHR_G_O/boreholeSampleDescription/bhrgcom:BoreholeSampleDescription/bhrgcom:descriptiveBoreholeLog/bhrgcom:DescriptiveBoreholeLog/bhrgcom:layer/bhrgcom:Layer"
layer = xml_tree.find(path_layer_element, namespaces=xml_tree.nsmap)
print(etree.tostring(layer, pretty_print=True, encoding="unicode"))
<bhrgcom:Layer xmlns:bhrgcom="http://www.broservices.nl/xsd/bhrgcommon/3.1" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:brocom="http://www.broservices.nl/xsd/brocommon/3.0" xmlns="http://www.broservices.nl/xsd/dsbhrg/3.1" gml:id="BRO_0006">
<bhrgcom:upperBoundary uom="m">0.000</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">waargenomen</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">0.250</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">zwakZandigeKlei</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">matigHumeus</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
<bhrgcom:particularConstituent>
<bhrgcom:constituentType codeSpace="urn:bro:bhrg:ParticularConstituentType">puin</bhrgcom:constituentType>
<bhrgcom:archiveClass codeSpace="urn:bro:bhrg:ArchiveClass">onbekend</bhrgcom:archiveClass>
</bhrgcom:particularConstituent>
</bhrgcom:soil>
</bhrgcom:Layer>
As we can see, additional attributes are available, such as “gravelContentClass”,
“carbonateContentClass”, and “colour”. These can easily be retrieved by adding them to the
list of “layer-attributes” in the schema and using the new schema as input to read the data.
Let’s use the updated schema to read the XML file again and check the data attribute of the
resulting Collection to verify that it works.
extra_attributes = ["gravelContentClass", "carbonateContentClass", "colour"]
bhrg_schema["data"]["layer-attributes"].extend(
extra_attributes
) # Add the list of extra attributes to the schema
bhrg = geost.read_bhrg(bhrg_file, schema=bhrg_schema)
bhrg.data
| nr | x | y | surface | end | top | bottom | soilNameNEN5104 | gravelContentClass | carbonateContentClass | colour | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | BHR000000396406 | 126148.999929 | 452161.999985 | 0.69 | -2.31 | 0.00 | 0.25 | zwakZandigeKlei | onbekend | onbekend | onbekend |
| 1 | BHR000000396406 | 126148.999929 | 452161.999985 | 0.69 | -2.31 | 0.25 | 1.60 | sterkSiltigeKlei | onbekend | onbekend | onbekend |
| 2 | BHR000000396406 | 126148.999929 | 452161.999985 | 0.69 | -2.31 | 1.60 | 2.00 | zwakZandigeKlei | onbekend | onbekend | onbekend |
| 3 | BHR000000396406 | 126148.999929 | 452161.999985 | 0.69 | -2.31 | 2.00 | 2.50 | sterkZandigeKlei | onbekend | onbekend | onbekend |
| 4 | BHR000000396406 | 126148.999929 | 452161.999985 | 0.69 | -2.31 | 2.50 | 3.00 | zwakSiltigZand | onbekend | onbekend | onbekend |
Even though the extra attributes have the value “onbekend” (i.e., unknown) for every layer in this particular borehole, we can see that they were successfully retrieved from the XML file.
Extending the schema#
So far, we have shown how to extend a schema to retrieve additional layer attributes. However, the schema can also be extended to retrieve other information that will be added to the header. Before we show how this can be done, it is worth taking a closer look at the XML file itself.
# We can print the XML tree that we loaded before
print(etree.tostring(xml_tree, pretty_print=True, encoding="unicode"))
<dispatchDataResponse xmlns:bhrgcom="http://www.broservices.nl/xsd/bhrgcommon/3.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:brocom="http://www.broservices.nl/xsd/brocommon/3.0" xmlns="http://www.broservices.nl/xsd/dsbhrg/3.1" xmlns:gml="http://www.opengis.net/gml/3.2">
<brocom:responseType>dispatch</brocom:responseType>
<brocom:requestReference>-</brocom:requestReference>
<brocom:dispatchTime>2025-08-04T11:36:40+02:00</brocom:dispatchTime>
<dispatchDocument>
<BHR_G_O gml:id="BRO_0013">
<brocom:broId>BHR000000396406</brocom:broId>
<brocom:deliveryAccountableParty>50200097</brocom:deliveryAccountableParty>
<brocom:qualityRegime>IMBRO/A</brocom:qualityRegime>
<deliveryContext codeSpace="urn:bro:bhrg:DeliveryContext">archiefoverdracht</deliveryContext>
<surveyPurpose codeSpace="urn:bro:bhrg:SurveyPurpose">onbekend</surveyPurpose>
<discipline codeSpace="urn:bro:bhrg:Discipline">geologie</discipline>
<surveyProcedure codeSpace="urn:bro:bhrg:SurveyProcedure">geen</surveyProcedure>
<researchReportDate>
<brocom:date>2004-09-22</brocom:date>
</researchReportDate>
<registrationHistory>
<brocom:objectRegistrationTime>2025-02-04T17:41:43+01:00</brocom:objectRegistrationTime>
<brocom:registrationStatus codeSpace="urn:bro:RegistrationStatus">voltooid</brocom:registrationStatus>
<brocom:registrationCompletionTime>2025-02-04T17:41:43+01:00</brocom:registrationCompletionTime>
<brocom:corrected>nee</brocom:corrected>
<brocom:underReview>nee</brocom:underReview>
<brocom:deregistered>nee</brocom:deregistered>
<brocom:reregistered>nee</brocom:reregistered>
</registrationHistory>
<NITGCode>B31G2129</NITGCode>
<reportHistory>
<bhrgcom:event>
<bhrgcom:date>
<brocom:date>2004-09-22</brocom:date>
</bhrgcom:date>
<bhrgcom:name codeSpace="urn:bro:bhrg:EventName">volledigGerapporteerd</bhrgcom:name>
</bhrgcom:event>
</reportHistory>
<deliveredLocation>
<bhrgcom:location>
<gml:Point srsName="urn:ogc:def:crs:EPSG::28992" gml:id="BRO_0001">
<gml:pos>126149.000 452162.000</gml:pos>
</gml:Point>
</bhrgcom:location>
<bhrgcom:horizontalPositioningDate>
<brocom:date>2004-09-22</brocom:date>
</bhrgcom:horizontalPositioningDate>
<bhrgcom:horizontalPositioningMethod codeSpace="urn:bro:bhrg:HorizontalPositioningMethod">onbekend</bhrgcom:horizontalPositioningMethod>
</deliveredLocation>
<deliveredVerticalPosition>
<bhrgcom:localVerticalReferencePoint codeSpace="urn:bro:bhrg:LocalVerticalReferencePoint">maaiveld</bhrgcom:localVerticalReferencePoint>
<bhrgcom:offset uom="m">0.690</bhrgcom:offset>
<bhrgcom:verticalDatum codeSpace="urn:bro:bhrg:VerticalDatum">NAP</bhrgcom:verticalDatum>
<bhrgcom:verticalPositioningDate>
<brocom:date>2004-09-22</brocom:date>
</bhrgcom:verticalPositioningDate>
<bhrgcom:verticalPositioningMethod codeSpace="urn:bro:bhrg:VerticalPositioningMethod">onbekend</bhrgcom:verticalPositioningMethod>
</deliveredVerticalPosition>
<standardizedLocation>
<brocom:location srsName="urn:ogc:def:crs:EPSG::4258" gml:id="BRO_0002">
<gml:pos>52.057009967 4.966538731</gml:pos>
</brocom:location>
<brocom:coordinateTransformation codeSpace="urn:bro:CoordinateTransformation">RDNAPTRANS2018</brocom:coordinateTransformation>
</standardizedLocation>
<boring>
<bhrgcom:Boring gml:id="BRO_0005">
<bhrgcom:boringStartDate>
<brocom:date>2004-09-22</brocom:date>
</bhrgcom:boringStartDate>
<bhrgcom:boringEndDate>
<brocom:voidReason>onbekend</brocom:voidReason>
</bhrgcom:boringEndDate>
<bhrgcom:preparation xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" codeSpace="urn:bro:bhrg:Preparation" xsi:nil="true"/>
<bhrgcom:trajectoryExcavated>nee</bhrgcom:trajectoryExcavated>
<bhrgcom:rockReached>nee</bhrgcom:rockReached>
<bhrgcom:boringProcedure codeSpace="urn:bro:bhrg:BoringProcedure">onbekend</bhrgcom:boringProcedure>
<bhrgcom:finalDepthBoring uom="m">3.00</bhrgcom:finalDepthBoring>
<bhrgcom:stopCriterion codeSpace="urn:bro:bhrg:StopCriterionField">onbekend</bhrgcom:stopCriterion>
<bhrgcom:samplingProcedure codeSpace="urn:bro:bhrg:SamplingProcedure">onbekend</bhrgcom:samplingProcedure>
<bhrgcom:finalDepthSampling xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uom="m" xsi:nil="true"/>
<bhrgcom:subsurfaceContaminated xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
<bhrgcom:boreholeCompleted>onbekend</bhrgcom:boreholeCompleted>
<bhrgcom:boredInterval>
<bhrgcom:BoredInterval gml:id="BRO_0003">
<bhrgcom:beginDepth uom="m">0.00</bhrgcom:beginDepth>
<bhrgcom:endDepth uom="m">3.00</bhrgcom:endDepth>
<bhrgcom:boringTechnique codeSpace="urn:bro:bhrg:BoringTechnique">handDraaien</bhrgcom:boringTechnique>
<bhrgcom:boredDiameter xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uom="mm" xsi:nil="true"/>
</bhrgcom:BoredInterval>
</bhrgcom:boredInterval>
<bhrgcom:sampledInterval>
<bhrgcom:SampledInterval gml:id="BRO_0004">
<bhrgcom:beginDepth uom="m">0.00</bhrgcom:beginDepth>
<bhrgcom:endDepth uom="m">3.00</bhrgcom:endDepth>
<bhrgcom:preTreatment codeSpace="urn:bro:bhrg:PreTreatment">onbekend</bhrgcom:preTreatment>
<bhrgcom:samplingMethod codeSpace="urn:bro:bhrg:SamplingMethod">opDiepteLosroeren</bhrgcom:samplingMethod>
<bhrgcom:samplingQuality codeSpace="urn:bro:bhrg:SamplingQuality">geroerd</bhrgcom:samplingQuality>
</bhrgcom:SampledInterval>
</bhrgcom:sampledInterval>
</bhrgcom:Boring>
</boring>
<boreholeSampleDescription>
<bhrgcom:BoreholeSampleDescription gml:id="BRO_0012">
<bhrgcom:descriptionReportDate>
<brocom:date>2004-09-22</brocom:date>
</bhrgcom:descriptionReportDate>
<bhrgcom:descriptionProcedure codeSpace="urn:bro:bhrg:DescriptionProcedure">NEN5104plusOnbekend</bhrgcom:descriptionProcedure>
<bhrgcom:descriptiveBoreholeLog>
<bhrgcom:DescriptiveBoreholeLog gml:id="BRO_0011">
<bhrgcom:descriptionQuality codeSpace="urn:bro:bhrg:DescriptionQuality">geologischNEN5104Archief</bhrgcom:descriptionQuality>
<bhrgcom:describedSamplesQuality codeSpace="urn:bro:bhrg:DescribedSamplesQuality">geroerd</bhrgcom:describedSamplesQuality>
<bhrgcom:continuouslySampled>ja</bhrgcom:continuouslySampled>
<bhrgcom:descriptionLocation codeSpace="urn:bro:bhrg:DescriptionLocation">veld</bhrgcom:descriptionLocation>
<bhrgcom:describedMaterial codeSpace="urn:bro:bhrg:DescribedMaterial">grond</bhrgcom:describedMaterial>
<bhrgcom:sampleMoistness codeSpace="urn:bro:bhrg:SampleMoistness">onbekend</bhrgcom:sampleMoistness>
<bhrgcom:layer>
<bhrgcom:Layer gml:id="BRO_0006">
<bhrgcom:upperBoundary uom="m">0.000</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">waargenomen</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">0.250</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">zwakZandigeKlei</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">matigHumeus</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
<bhrgcom:particularConstituent>
<bhrgcom:constituentType codeSpace="urn:bro:bhrg:ParticularConstituentType">puin</bhrgcom:constituentType>
<bhrgcom:archiveClass codeSpace="urn:bro:bhrg:ArchiveClass">onbekend</bhrgcom:archiveClass>
</bhrgcom:particularConstituent>
</bhrgcom:soil>
</bhrgcom:Layer>
</bhrgcom:layer>
<bhrgcom:layer>
<bhrgcom:Layer gml:id="BRO_0007">
<bhrgcom:upperBoundary uom="m">0.250</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">1.600</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">sterkSiltigeKlei</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">zwakHumeus</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
</bhrgcom:soil>
</bhrgcom:Layer>
</bhrgcom:layer>
<bhrgcom:layer>
<bhrgcom:Layer gml:id="BRO_0008">
<bhrgcom:upperBoundary uom="m">1.600</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">2.000</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">zwakZandigeKlei</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">zwakHumeus</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
</bhrgcom:soil>
</bhrgcom:Layer>
</bhrgcom:layer>
<bhrgcom:layer>
<bhrgcom:Layer gml:id="BRO_0009">
<bhrgcom:upperBoundary uom="m">2.000</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">2.500</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">sterkZandigeKlei</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">zwakHumeus</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
</bhrgcom:soil>
</bhrgcom:Layer>
</bhrgcom:layer>
<bhrgcom:layer>
<bhrgcom:Layer gml:id="BRO_0010">
<bhrgcom:upperBoundary uom="m">2.500</bhrgcom:upperBoundary>
<bhrgcom:upperBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">onbekend</bhrgcom:upperBoundaryDetermination>
<bhrgcom:lowerBoundary uom="m">3.000</bhrgcom:lowerBoundary>
<bhrgcom:lowerBoundaryDetermination codeSpace="urn:bro:bhrg:BoundaryPositioningMethod">voorbepaald</bhrgcom:lowerBoundaryDetermination>
<bhrgcom:anthropogenic>onbekend</bhrgcom:anthropogenic>
<bhrgcom:rooted>onbekend</bhrgcom:rooted>
<bhrgcom:soil>
<bhrgcom:soilNameNEN5104 codeSpace="urn:bro:bhrg:SoilNameNEN5104">zwakSiltigZand</bhrgcom:soilNameNEN5104>
<bhrgcom:gravelContentClass codeSpace="urn:bro:bhrg:GravelContentClass">onbekend</bhrgcom:gravelContentClass>
<bhrgcom:organicMatterContentClassNEN5104 codeSpace="urn:bro:bhrg:OrganicMatterContentClassNEN5104">onbekend</bhrgcom:organicMatterContentClassNEN5104>
<bhrgcom:carbonateContentClass codeSpace="urn:bro:bhrg:CarbonateContentClass">onbekend</bhrgcom:carbonateContentClass>
<bhrgcom:colour codeSpace="urn:bro:bhrg:Colour">onbekend</bhrgcom:colour>
<bhrgcom:sandFraction>
<bhrgcom:sandMedianClass codeSpace="urn:bro:bhrg:SandMedianClass">matigFijnNEN5104</bhrgcom:sandMedianClass>
<bhrgcom:sandSorting codeSpace="urn:bro:bhrg:SandSorting">onbekend</bhrgcom:sandSorting>
</bhrgcom:sandFraction>
</bhrgcom:soil>
</bhrgcom:Layer>
</bhrgcom:layer>
</bhrgcom:DescriptiveBoreholeLog>
</bhrgcom:descriptiveBoreholeLog>
</bhrgcom:BoreholeSampleDescription>
</boreholeSampleDescription>
</BHR_G_O>
</dispatchDocument>
</dispatchDataResponse>
By inspecting the XML tree, we can see where specific elements can be found and how these relate to the “xpath” keys in the schema. The indentation level of the XML tree represents the hierarchy of so-called “elements” and “child-elements”.
The geost.read_bhrg function begins searching for the data from the <dispatchDocument>
element because the schema defines this as the “payload_root”. This element can contain one or
more child-elements that hold the actual borehole data: <BHR_G_O gml:id="BRO_xxxx"> elements.
When “payload_root” is not defined in an input schema, the function begins searching from the
root element of the tree (“dispatchDataResponse”), which may lead to unexpected results. We
can see that the ID of the borehole is stored in the <brocom:broId> element and that this
was used as the “xpath” entry to retrieve “nr” as the borehole ID.
We can extend the schema to retrieve additional header information by adding an extra
dictionary with at least an “xpath” entry and, if needed, entries for “resolver” and/or
“el-attr”. If we look at the complete XML tree above, we can see that it contains a
<registrationHistory> element with child-elements containing registration dates for this
BHR-G object. Let’s assume for now that this information is relevant to us and that we
therefore want to retrieve it as additional header information. We can achieve this by
extending the schema as shown below. We will first retrieve the data by specifying where it
is located in the XML tree and then read the file again.
date = {"xpath": "registrationHistory/brocom:registrationCompletionTime"}
bhrg_schema["date"] = date
bhrg = geost.read_bhrg(bhrg_file, schema=bhrg_schema)
print(bhrg.header) # Print the header to see the new date attribute
print(
f"\nDtype of date: {bhrg.header['date'].dtypes}"
) # Print the data types of the header attributes
nr surface end date x \
0 BHR000000396406 0.69 -2.31 2025-02-04T17:41:43+01:00 126148.999929
y geometry
0 452161.999985 POINT (126149 452162)
Dtype of date: str
Adding your own resolver#
The date has now been nicely added to the header of the Collection. However, the
resulting data type of the date is “object”, which is a “string-like” data type in Pandas
DataFrames. Since this is a date, it may be more convenient to represent it directly as a
“datetime” type. We can achieve this by adding our own “resolver” function that translates
the text attribute of the retrieved date element into a datetime type. Let’s write our
helper function and update the schema accordingly.
import pandas as pd
def to_datetime(date_str):
"""Convert a date string to a pandas datetime object."""
return pd.to_datetime(date_str)
date = {
"xpath": "registrationHistory/brocom:registrationCompletionTime",
"resolver": to_datetime,
"el-attr": "text", # This ensures we use the text content of the element
}
bhrg_schema["date"] = date
bhrg = geost.read_bhrg(bhrg_file, schema=bhrg_schema)
print(bhrg.header) # Print the header to see the new date attribute
print(
f"\nDtype of date: {bhrg.header['date'].dtypes}"
) # Print the data types of the header attributes
nr surface end date x \
0 BHR000000396406 0.69 -2.31 2025-02-04 17:41:43+01:00 126148.999929
y geometry
0 452161.999985 POINT (126149 452162)
Dtype of date: datetime64[us, UTC+01:00]
Now the data type is “datetime64[ns, UTC+01:00]”, which is more convenient for operations involving date selections, for example. Of course, changing the data type of a date could also have been done in other ways, without adding a resolver. However, in some cases, the data in elements needs to be processed in such specific ways that adding our own resolvers is necessary.
Very specific needs#
Sometimes, the requirements may become so specific that GeoST can no longer load the data
into Collection objects. This may occur when schemas become too specific, for example when
resolvers return complex data structures that cannot be easily added as rows to DataFrames.
This is not necessarily a problem, because GeoST still provides functionality to parse the
XML files. However, instead of returning a Collection object, the parser returns a
dictionary containing the retrieved data according to the schema.
Let’s assume for now that the bhrg_schema has become too specific and that the standard
reader functions are no longer sufficient. The example below shows how you can still use
GeoST to retrieve the data for further use in your own application, or continue processing
it into a format that is compatible with GeoST objects.
data = xml.read_bhrg(
bhrg_file, schema=bhrg_schema
) # bhrg_schema is now our "too specific" schema
pprint(data, sort_dicts=False)
{'nr': 'BHR000000396406',
'location': (52.057009967, 4.966538731),
'crs': 'urn:ogc:def:crs:EPSG::4258',
'surface': 0.69,
'vertical_datum': 'NAP',
'end': 3.0,
'data': defaultdict(<class 'list'>,
{'upperBoundary': ['0.000',
'0.250',
'1.600',
'2.000',
'2.500'],
'lowerBoundary': ['0.250',
'1.600',
'2.000',
'2.500',
'3.000'],
'soilNameNEN5104': ['zwakZandigeKlei',
'sterkSiltigeKlei',
'zwakZandigeKlei',
'sterkZandigeKlei',
'zwakSiltigZand'],
'gravelContentClass': ['onbekend',
'onbekend',
'onbekend',
'onbekend',
'onbekend'],
'carbonateContentClass': ['onbekend',
'onbekend',
'onbekend',
'onbekend',
'onbekend'],
'colour': ['onbekend',
'onbekend',
'onbekend',
'onbekend',
'onbekend']}),
'date': Timestamp('2025-02-04 17:41:43+0100', tz='UTC+01:00')}
As you can see, a clean dictionary is still returned, containing the data for each key you have added to your schema, and you can continue working with the data. As mentioned at the beginning of this tutorial, reading XML files for the other data sources (BHR-GT, BHR-P, CPT, and SFR) works in exactly the same way and offers the same possibilities.