Folder Structure

Overview

This document outlines the standardized folder structure for sensor data storage within the iRODS data platform. The structure is designed to organize research data hierarchically, from institutional level down to individual data files, ensuring consistent and clear organization across all research projects.

Interactive Path Generator

Use our interactive tool to generate standardized file paths and folder structures for your data:

Delta Enigma Path Generator

Use this tool to generate standardized file paths and folder structures for your data.

Institution code:

Work package:

Location:

Sensor type:

Sensor number:

Data type: raw-data processed-data published-data

File extension:

Date:

Generate Path

Generated Path Information

Folder Structure Details

Complete Path Example

Research/
└── research-uu/
    └── WP3/
          └── zandmotor/
              └── raw-data/
                  └── camera/
                      └── camera1/
                          └── 2024/
                              └── May.26/
                                  └── May.26.2024.zandmotor.camera1.raw-data.jpg

Hierarchy Levels

1. Root Level

   • Research/: Top-level directory for all research data

2. Institution Level

   • research-uu/: Institution-specific directory

   • Format: research-{institution-code}

3. Work Package Level

   • WP3/: Work package identifier

   • Format: WP{number}

4. Location Level

   • zandmotor/: Geographic or site-specific location

   • Format: {location-name}

5. Data Classification

   • Distinguishes between different data processing stages

   • Available categories:

     • raw-data/: Unprocessed data directly from sensors

     • processed-data/: Data that has undergone processing or analysis

     • published-data/: Final, published versions of datasets

6. Sensor Type

   • camera/: Type of sensor or data collection device

   • Format: {sensor-type}

7. Device Instance

   • camera1/: Specific device identifier

   • Format: {sensor-type}{number}

8. Temporal Organization

   • 2024/: Year

   • 147_May.26/: Day folder

   • Format: {day_of_year}_{month}.{day}

File Naming Convention

Pattern

{unix_timestamp}.{human_readable_datetime}.{location}.{device_id}.{data_type}.{extension}

Example

1716706803.Sun.May.26_07_00_03.UTC.2024.zandmotor.camera1.raw-data.jpg

Components

• 1716706803: Unix timestamp

• Sun.May.26_07_00_03.UTC.2024: Human readable date and time

• zandmotor: Location identifier (lowercase)

• camera1: Device identifier (sensor type + number)

• raw-data: Processing stage

• jpg: File extension (lowercase)

Implementation Options

Web Tool

The interactive web tool (shown above) provides the easiest way to generate paths and filenames. It’s ideal for:

• Quick path generation

• Interactive exploration of the structure

• Immediate feedback on generated paths

Python Script

For programmatic or command-line usage, a Python script is available:

1. Download: generate_path.py

2. Run in interactive mode:

   python3 generate_path.py --interactive
   

3. Follow the prompts to enter your data

Features:

• Interactive prompts with sensible defaults

• Date validation

• Option to create directory structure

• Both interactive and non-interactive modes

• Clear output of generated paths and filenames

Best Practices

1. Folder Names

   • Use lowercase letters

   • Use hyphens for spaces

   • Follow the hierarchical structure strictly

2. File Names

   • Include all required components

   • Use consistent formatting

   • Follow the established pattern

3. Data Organization

   • Keep raw data separate from processed data

   • Use appropriate data classification

   • Maintain consistent temporal organization