Usage Guide

This guide covers detailed usage patterns and advanced features of dartfx-dataverse.

Server Management

Creating Server Connections

The DataverseServer class is the main entry point for interacting with Dataverse installations:

from dartfx.dataverse import DataverseServer, ServerInstallation

# Create a server installation object
installation = ServerInstallation(
    name="My Dataverse",
    hostname="dataverse.example.com",
    description="Example Dataverse installation"
)

# Create server connection
server = DataverseServer(installation)

Configuration Options

The DataverseServer class accepts several configuration parameters:

server = DataverseServer(
    server=installation,
    api_key="your-api-key",          # Optional API key
    ssl_verify=True,                  # SSL certificate verification
    on_api_error="raise",             # Error handling: "raise" or "none"
    on_api_success_return="json"      # Response format: "json", "text", or "response"
)

Custom Session Configuration

You can provide a custom requests session for advanced configuration:

import requests_cache
from datetime import timedelta

# Create custom cached session
session = requests_cache.CachedSession(
    cache_name='dataverse_cache',
    backend='sqlite',
    expire_after=timedelta(hours=2),
    allowable_methods=['GET', 'POST'],
    stale_if_error=True
)

server = DataverseServer(
    server=installation,
    session=session
)

Server Information

Retrieve server metadata and version information:

# Get server information
info = server.get_server_info()
print(f"Version: {info['data']['version']}")
print(f"Build: {info['data']['build']}")

# Get metadata blocks
metadata_blocks = server.get_metadatablocks()
for block in metadata_blocks['data']:
    print(f"- {block['name']}: {block.get('displayName', 'N/A')}")

Searching

Simple Search

For quick searches, use the search_simple method:

# Simple text search
results = server.search_simple("climate")

# Search with pagination
results = server.search_simple("climate", start=10, per_page=20)

Advanced Search with Parameters

Use SearchParameters for full control over search options:

from dartfx.dataverse import SearchParameters

params = SearchParameters(
    q="title:climate AND description:temperature",
    type="dataset",
    sort="date",
    order="desc",
    per_page=50,
    start=0,
    show_relevance=True,
    show_facets=True,
    show_entity_ids=False
)

results = server.search(params)

Search Query Syntax

The search query supports Solr query syntax:

# Field-specific search
params = SearchParameters(q="title:climate")

# Boolean operators
params = SearchParameters(q="climate AND temperature")
params = SearchParameters(q="climate OR weather")
params = SearchParameters(q="climate NOT politics")

# Phrase search
params = SearchParameters(q='"climate change"')

# Wildcard search
params = SearchParameters(q="climat*")
params = SearchParameters(q="*climate*")

# Range search
params = SearchParameters(
    q="*",
    fq=["publicationDate:[2020 TO 2024]"]
)

Filtering Results

Use filter queries (fq) to narrow results:

params = SearchParameters(
    q="*",
    type="dataset",
    fq=[
        "publicationDate:[2020 TO *]",        # Published after 2020
        "authorName:Smith",                   # Author is Smith
        "dvName:climateData"                  # In climate dataverse
    ]
)

Faceted Search

Enable facets to see result distributions:

params = SearchParameters(
    q="climate",
    show_facets=True
)

results = server.search(params)

# Process facets
if 'facets' in results['data']:
    for facet in results['data']['facets']:
        print(f"\nFacet: {facet['friendly_name']}")
        for label in facet['labels']:
            print(f"  {label['label']}: {label['count']}")

Geographic Search

Search by geographic location:

# Search within radius of a point
params = SearchParameters(
    q="*",
    geo_point="42.3601,-71.0589",  # Boston, MA (lat,lon)
    geo_radius="50"                 # 50 km radius
)

results = server.search(params)

Searching Multiple Types

Search across dataverses, datasets, and files:

# Search all types
params = SearchParameters(
    q="climate",
    type=["dataverse", "dataset", "file"]
)

results = server.search(params)

Pagination

Handle large result sets with pagination:

def paginate_search(server, query, items_per_page=100):
    """Generator function for paginating through search results."""
    start = 0

    while True:
        params = SearchParameters(
            q=query,
            per_page=items_per_page,
            start=start
        )

        results = server.search(params)
        items = results['data']['items']

        if not items:
            break

        for item in items:
            yield item

        start += items_per_page

# Usage
for item in paginate_search(server, "climate"):
    print(item['name'])

Dataset Metadata & Export Formats

Retrieve detailed metadata and standard export formats for datasets using their Persistent Identifier (DOI or Handle).

Supported Metadata Formats & Standards

Exporter / Format	Output Encoding	Extension	Primary Domain & Use Case
Native JSON	JSON	`.dataverse.json`	Full Dataverse dataset metadata blocks, file manifests, terms, and versions.
Croissant ML (`croissant`)	JSON-LD (schema.org/cr)	`.croissant.json`	Machine Learning pipelines, Hugging Face, PyTorch, automated training sets.
DDI Codebook 2.5 (`ddi`)	XML	`.ddi-c.xml`	Social sciences, variable-level codebooks, statistical documentation.
Schema.org (`schema.org`)	JSON-LD (schema.org)	`.schema.json`	Search engines, Google Dataset Search, web schema indexing.
DataCite (`datacite`)	XML	`.datacite.xml`	DOI registration, persistent identifier tracking, academic citations.
Dublin Core (`oai_dc`)	XML	`.xml`	Cross-domain digital library metadata exchange.

Getting Dataset Metadata (Native JSON)

The get_dataset method returns the full metadata for a dataset in JSON format:

# Retrieve dataset by DOI
dataset = server.get_dataset("doi:10.5683/SP3/FNS9EF")

# Accessing citation and metadata blocks
version = dataset['data']['latestVersion']
for block_name, block in version['metadataBlocks'].items():
    print(f"Block: {block_name}")
    for field in block['fields']:
        print(f"  - {field['typeName']}: {field['value']}")

Getting Standard Export Formats

Use the get_dataset_export method with the target exporter identifier:

pid = "doi:10.5683/SP3/FNS9EF"

# 1. Croissant ML (JSON-LD)
croissant_str = server.get_dataset_export(pid, exporter="croissant")

# 2. DDI Codebook 2.5 (XML)
ddi_xml = server.get_dataset_export(pid, exporter="ddi")

# 3. Schema.org (JSON-LD)
schema_json = server.get_dataset_export(pid, exporter="schema.org")

# 4. DataCite (XML)
datacite_xml = server.get_dataset_export(pid, exporter="datacite")

# 5. Dublin Core (XML)
dc_xml = server.get_dataset_export(pid, exporter="oai_dc")

Listing Available Exporters on Server

You can inspect the complete list of export formats supported and enabled on any given Dataverse server:

formats = server.get_info_export_formats()
for format_name, details in formats['data'].items():
    print(f"{format_name}: {details['displayName']} ({details['mediaType']})")

Metadata Harvester Subsystem

The toolkit includes an incremental metadata harvester subsystem for bulk discovery, repository profiling, and synchronization.

Incremental Server Synchronization

Use ServerHarvester to synchronize datasets across one or more metadata formats into a structured directory hierarchy:

from pathlib import Path
from dartfx.dataverse import ServerHarvester

harvester = ServerHarvester(
    server_dir=Path("./harvested_data/dataverse.harvard.edu"),
    host="dataverse.harvard.edu",
    verbose=True,
    dry_run=False
)

# Sync Croissant, Native JSON, and DDI metadata for rectangular tabular datasets
summary = harvester.sync(
    formats=["croissant", "native", "ddi"],
    query="climate",
    limit=25,
    tabular_only=True
)

print(f"Processed: {summary['datasets_count']} datasets")
print(f"Added: {summary['added']}, Updated: {summary['updated']}, Unchanged: {summary['unchanged']}")

Repository Statistics & Profiling

Query live or cached dataset counts, total file counts, and tabular rectangular data file counts:

from dartfx.dataverse import fetch_server_stats

stats = fetch_server_stats("dataverse.harvard.edu")
print(f"Host: {stats['host']} (Version: {stats['server_version']})")
print(f"Total Datasets: {stats['datasets']:,}")
print(f"Total Files: {stats['total_files']:,}")
print(f"Tabular Files: {stats['tabular_files']:,} ({stats['tabular_pct']}%)")

Harvest Error Analysis

Scan local harvest manifests to classify and aggregate error types across repositories:

from pathlib import Path
from dartfx.dataverse import analyze_harvest_errors

errors_report = analyze_harvest_errors(Path("./harvested_data"))
print(f"Total Failed Records: {errors_report['total_errors']}")
for category, count in errors_report['by_category'].items():
    print(f"  {category}: {count}")

Token Resolution & Management

Manage and resolve per-server API tokens automatically:

from dartfx.dataverse import resolve_server_token, save_server_token

# Resolve token from env vars, .api_token files, or .dataverse_tokens.json
token = resolve_server_token("dataverse.unc.edu")

# Persist token for future runs
save_server_token("dataverse.unc.edu", "my-secret-token")

Error Handling

Understanding Errors

The package provides the DataverseApiError exception for API-related errors:

from dartfx.dataverse import DataverseApiError

try:
    results = server.search_simple("test")
except DataverseApiError as e:
    print(f"Error: {e.message}")
    print(f"URL: {e.url}")
    print(f"Status Code: {e.status_code}")

    # Access the raw response if needed
    if e.response:
        print(f"Response Text: {e.response.text}")

Error Handling Modes

Configure how the server handles API errors:

# Raise exceptions on errors (default)
server = DataverseServer(
    server=installation,
    on_api_error="raise"
)

# Return None on errors (silent mode)
server = DataverseServer(
    server=installation,
    on_api_error="none"
)

result = server.get_server_info()
if result is None:
    print("Error occurred, but no exception was raised")

Retry Logic

Implement retry logic for transient failures:

from time import sleep

def search_with_retry(server, query, max_retries=3):
    """Search with exponential backoff retry."""
    for attempt in range(max_retries):
        try:
            return server.search_simple(query)
        except DataverseApiError as e:
            if e.status_code >= 500 and attempt < max_retries - 1:
                wait_time = 2 ** attempt
                print(f"Retry {attempt + 1}/{max_retries} in {wait_time}s...")
                sleep(wait_time)
            else:
                raise

results = search_with_retry(server, "climate")

Best Practices

Use Caching

Enable caching to reduce API calls and improve performance:

import requests_cache

# Cache responses for 1 hour
session = requests_cache.CachedSession(
    expire_after=3600,
    allowable_methods=['GET']
)

server = DataverseServer(server=installation, session=session)

Rate Limiting

Be respectful of server resources:

from time import sleep

def search_with_rate_limit(server, queries, delay=1.0):
    """Search multiple queries with rate limiting."""
    results = []

    for query in queries:
        result = server.search_simple(query)
        results.append(result)
        sleep(delay)  # Wait between requests

    return results

Validate Input

Use Pydantic models to validate input data:

from dartfx.dataverse import SearchParameters
from pydantic import ValidationError

try:
    # This will raise ValidationError if invalid
    params = SearchParameters(
        q="test",
        per_page=2000  # Exceeds maximum of 1000
    )
except ValidationError as e:
    print(f"Invalid parameters: {e}")

Handle Missing Data

Always check for optional fields in responses:

results = server.search_simple("test")

for item in results['data']['items']:
    name = item.get('name', 'Unnamed')
    description = item.get('description', 'No description available')
    published_at = item.get('published_at', 'Not published')

    print(f"{name}: {description} (Published: {published_at})")

Working with Multiple Installations

Comparing Results Across Servers

from dartfx.dataverse import fetch_dataverse_installations, DataverseServer

# Get installations
installations = fetch_dataverse_installations()

# Filter for active installations
active = [i for i in installations if i.hostname]

# Search across multiple servers
query = "open data"
results_by_server = {}

for installation in active[:5]:  # Limit to first 5
    try:
        server = DataverseServer(installation=installation)
        results = server.search_simple(query)
        results_by_server[installation.name] = results['data']['total_count']
    except Exception as e:
        print(f"Error with {installation.name}: {e}")

# Display results
for name, count in sorted(results_by_server.items(), key=lambda x: x[1], reverse=True):
    print(f"{name}: {count} results")

Advanced Topics

Custom User Agent

Set a custom user agent for your requests:

server = DataverseServer(installation)
server.user_agent = "MyApp/1.0 (contact@example.com)"

Response Format Options

Control the response format:

# Return JSON (default)
server = DataverseServer(
    server=installation,
    on_api_success_return="json"
)

# Return raw text
server = DataverseServer(
    server=installation,
    on_api_success_return="text"
)

# Return response object
server = DataverseServer(
    server=installation,
    on_api_success_return="response"
)

response = server.get_server_info()
print(response.status_code)
print(response.headers)

Command Line Interface

The toolkit includes a full-featured command-line interface dartfx-dataverse for discovery, search, dataset export, repository statistics, and bulk metadata harvesting.

Tip

For full command reference, parameter options, JSON/CSV streaming examples, and environment configuration, see the comprehensive Command Line Interface (CLI) guide and Harvester Subsystem & CLI Utility (dartfx-dataverse harvest) guide.

Installations

List worldwide Dataverse installations:

dartfx-dataverse installations --limit 10

Search

Search for dataverses, datasets, and files:

# Default (Harvard Dataverse)
dartfx-dataverse search "climate change"

# Specific server and limit
dartfx-dataverse search "physics" --hostname demo.dataverse.org --per-page 5

# Filter by type
dartfx-dataverse search "biology" --type dataset

# Sort and order
dartfx-dataverse search "climate" --sort date --order desc

Server Info

Get information about a specific Dataverse server:

dartfx-dataverse info dataverse.harvard.edu

Metadata Blocks

List available metadata blocks for a server:

dartfx-dataverse metadatablocks dataverse.harvard.edu

Dataset Metadata

Retrieve dataset metadata or exports:

# Get JSON metadata
dartfx-dataverse dataset doi:10.5683/SP3/FNS9EF --hostname borealisdata.ca

# Get specific export format
dartfx-dataverse dataset doi:10.5683/SP3/FNS9EF -H borealisdata.ca --export ddi

Output Formats

All commands support a --format option (table, json, or csv):

dartfx-dataverse installations --format csv
dartfx-dataverse search "climate" --format json

API Key

Commands that require authentication (or to avoid rate limits) can use the --api-key option or the DATAVERSE_API_KEY environment variable:

export DATAVERSE_API_KEY="your-api-key"
dartfx-dataverse search "confidential data"

Debugging

Enable logging to debug issues:

import logging

# Enable debug logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger('dartfx.dataverse')
logger.setLevel(logging.DEBUG)

# Now all API calls will be logged
results = server.search_simple("test")