dartfx-dataverse

User Guide

  • Installation
    • Requirements
    • Development Installation (Current Method)
      • Using Hatch (Recommended for Development)
    • Dependencies
      • Core Dependencies
      • Optional Dependencies
        • Development Tools ([dev])
        • Documentation Tools ([docs])
      • Installing with Optional Dependencies
    • Future PyPI Installation
      • Using uv (Recommended)
      • Using pip
    • Verifying Installation
    • Troubleshooting
      • Import Errors
      • SSL Certificate Errors
    • Upgrading
    • Uninstalling
  • Quickstart Guide
    • Installation
    • Basic Workflow
    • Step 1: Discover Dataverse Installations
    • Step 2: Connect to a Server
      • With API Key (Optional)
    • Step 3: Search for Data
      • Advanced Search
    • Step 4: Filter and Refine
    • Working with Multiple Servers
    • Step 5: Retrieve Multi-Standard Dataset Metadata
    • Step 6: Incremental Harvesting & Statistics
    • Handling Errors
    • Configuring Caching
    • Next Steps
    • Common Use Cases
      • Finding Datasets by Subject
      • Searching Within a Collection
      • Geographic Search
    • Getting Help
  • Usage Guide
    • Server Management
      • Creating Server Connections
      • Configuration Options
      • Custom Session Configuration
      • Server Information
    • Searching
      • Simple Search
      • Advanced Search with Parameters
      • Search Query Syntax
      • Filtering Results
      • Faceted Search
      • Geographic Search
      • Searching Multiple Types
      • Pagination
    • Dataset Metadata & Export Formats
      • Supported Metadata Formats & Standards
      • Getting Dataset Metadata (Native JSON)
      • Getting Standard Export Formats
      • Listing Available Exporters on Server
    • Metadata Harvester Subsystem
      • Incremental Server Synchronization
      • Repository Statistics & Profiling
      • Harvest Error Analysis
      • Token Resolution & Management
    • Error Handling
      • Understanding Errors
      • Error Handling Modes
      • Retry Logic
    • Best Practices
      • Use Caching
      • Rate Limiting
      • Validate Input
      • Handle Missing Data
    • Working with Multiple Installations
      • Comparing Results Across Servers
    • Advanced Topics
      • Custom User Agent
      • Response Format Options
    • Command Line Interface
      • Installations
      • Search
      • Server Info
      • Metadata Blocks
      • Dataset Metadata
      • Output Formats
      • API Key
      • Debugging
  • Command Line Interface (CLI)
    • Overview & Syntax
    • Available Commands
    • Environment Variables & Configuration
    • 1. installations - List Global Repositories
      • Syntax
      • Options
      • Examples
    • 2. info - Server & Export Formats Inspection
      • Syntax
      • Arguments & Options
      • Output
      • Examples
    • 3. search - Search Datasets, Dataverses & Files
      • Syntax
      • Arguments & Options
      • Search Query Syntax Examples
      • Examples
    • 4. dataset - Metadata & Export Formats
      • Syntax
      • Arguments & Options
      • Examples
    • 5. metadatablocks - Metadata Schema Blocks
      • Syntax
      • Arguments & Options
      • Examples
    • 6. stats - Global Repository Statistics
      • Syntax
      • Options
      • Examples
    • 7. harvest - Metadata Harvester & Sync Engine
      • Syntax
      • Key Features
    • 8. errors - Harvest Error Analysis & Counts by Type
      • Syntax
      • Arguments & Options
      • Error Categories Detected
      • Examples
  • Harvester Subsystem & CLI Utility (dartfx-dataverse harvest)
    • 🚀 How to Use
      • Prerequisites & Installation
      • CLI Command Syntax
      • Options & Arguments
      • Common Usage Examples
        • 1. Harvest a Specific Server
        • 2. Filter Global Repositories by Country
        • 3. Targeted Keyword Search & Dry-Run Preview
        • 4. Quick Testing with Record Limit (--limit / -n)
        • 5. Harvest Only Tabular/Rectangular Data Datasets (--tabular / -t)
        • 6. Harvest Multiple Metadata Formats (--format / -f)
        • 7. Incremental Time-Bounded Sync
        • 8. Listing Available Dataverse Servers (CLI & cURL)
        • 9. Reporting Repository Statistics (--stats / --server-stats)
    • ⚙️ How It Works
      • Architectural Workflow
      • Local Directory Structure & Manifests
        • Manifest File (.manifest.json)
      • Intelligent Incremental Sync Logic
      • Multi-Tabular Dataset Exports & Metadata Standards
        • 1. Dataset-Level Export Granularity (One Document per Dataset)
        • 2. Cross-Standard Comparison Matrix
        • 3. Structural Examples
      • Understanding Metadata Validation & Errors
        • Croissant ML Validation Exceptions (['md5', 'sha256'] missing)
      • Understanding Server Access, API Tokens & Bot Protection
        • 1. Repositories Requiring API Token Authentication (HTTP 401 / :SearchApiRequiresToken)
        • 2. Bot Protection Interstitials & WAFs (HTTP 403 / HTTP 200 HTML)
        • 3. Legacy Directory Hostnames (HTTP 404)
      • Analyzing Harvest Errors (dartfx-dataverse errors)
        • Common Harvest Error Types
    • 🐍 Programmatic Python API
      • 1. Incremental Harvesting with ServerHarvester
      • 2. Fetch Active Datasets Catalog with fetch_active_datasets
      • 3. Query Server Statistics with fetch_server_stats
      • 4. Analyze Manifest Errors with analyze_harvest_errors
  • Examples
    • Example 1: Find All Climate Datasets
    • Example 2: Compare Datasets Across Installations
    • Example 3: Download Dataset Metadata
    • Example 4: Geographic Search for Environmental Data
    • Example 5: Build a Dataset Catalog
    • Example 6: Find Datasets by Author
    • Example 7: Monitor New Datasets
    • Example 8: Subject-Based Analysis
    • Example 9: Batch Export Dataset Information
    • Example 10: Create a Simple Search Interface
    • Example 11: Multi-Format Metadata Retrieval & Inspection
    • Example 12: Programmatic Harvesting & Error Classification
    • More Examples

API Reference

  • API Reference
    • Overview
    • Quick Links
      • Main Classes
      • Main Functions
    • Complete API
      • Server Classes
        • DataverseServer
        • ServerInstallation
        • Functions
        • Examples
      • Search Functionality
        • SearchParameters
        • Query Examples
        • Response Format
      • Data Models
        • Overview
        • ServerInstallation
        • SearchParameters
        • Working with Models
        • Type Hints
        • Optional vs Required Fields
        • Field Constraints
        • Best Practices
      • Exceptions
        • DataverseApiError
        • Common HTTP Status Codes
        • Usage Examples
        • Best Practices
        • Related
      • Harvester Module
        • ServerHarvester
        • Functions

Development

  • Contributing
    • Getting Started
    • Development Workflow
      • Making Changes
      • Code Style
    • Writing Tests
      • Test Structure
      • Running Tests
    • Documentation
      • Writing Documentation
      • Docstring Format
    • Submitting Changes
      • Pull Request Process
      • Pull Request Checklist
    • Code Review
      • What We Look For
      • Review Process
    • Reporting Bugs
      • Bug Report Guidelines
    • Feature Requests
    • Community Guidelines
      • Code of Conduct
      • Communication
    • Getting Help
    • Development Tips
      • Useful Commands
      • Debugging Tests
      • Working with Virtual Environments
    • Release Process
      • For Maintainers
    • Thank You!
  • Changelog
    • [0.2.0] - 2026-08-20
      • Added
    • [0.1.0] - 2026-03-11
      • Added
      • Changed
      • Fixed
    • Planned Features
      • v0.3.0+
      • v1.0.0
    • Version Support
      • Python Version Support
      • Dataverse Version Support
    • Contributing
    • Links
dartfx-dataverse
  • Search

© Copyright 2024-2026, Pascal Heus (Data Artifex).

Built with Sphinx using a theme provided by Read the Docs.