Quickstart Guide

This guide will help you get started with dartfx-dataverse in just a few minutes.

Installation

First, install the package using uv (recommended):

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dartfx-dataverse
uv pip install dartfx-dataverse

Or using pip:

pip install dartfx-dataverse

Basic Workflow

The typical workflow for using this package involves:

  1. Discovering available Dataverse installations (optional)

  2. Creating a server connection

  3. Performing searches or retrieving information

  4. Processing the results

Step 1: Discover Dataverse Installations

Get a list of all known Dataverse installations worldwide. This functionality leverages the community-maintained dataverse-installations project:

from dartfx.dataverse import fetch_dataverse_installations

# Fetch all known installations
installations = fetch_dataverse_installations()

# Display the first 5 installations
for installation in installations[:5]:
    print(f"{installation.name}")
    print(f"  Hostname: {installation.hostname}")
    print(f"  Country: {installation.country}")
    print(f"  Launch Year: {installation.launch_year}")
    print()

Step 2: Connect to a Server

Create a connection to a specific Dataverse server:

from dartfx.dataverse import DataverseServer, ServerInstallation

# Create server installation object
harvard = ServerInstallation(
    name="Harvard Dataverse",
    hostname="dataverse.harvard.edu"
)

# Create server connection
server = DataverseServer(installation=harvard)

# Get server information
info = server.get_server_info()
print(f"Server version: {info['data']['version']}")

With API Key (Optional)

If you have an API key for authenticated requests:

server = DataverseServer(
    installation=harvard,
    api_key="your-api-key-here"
)

Step 3: Search for Data

Perform a simple search:

# Simple text search
results = server.search_simple("climate")

print(f"Found {results['data']['total_count']} results")

# Display first 5 results
for item in results['data']['items'][:5]:
    print(f"- {item['name']}")

Step 4: Filter and Refine

Use filters to refine your search:

# Search with filters
params = SearchParameters(
    q="*",                           # Match all
    type="dataset",
    fq=[                             # Filter queries
        "publicationDate:[2020 TO *]",  # From 2020 onwards
        "dvName:climate"                # In climate dataverse
    ],
    per_page=20
)

results = server.search(params)

Working with Multiple Servers

You can work with multiple Dataverse installations simultaneously:

from dartfx.dataverse import DataverseServer, fetch_dataverse_installations

# Get installations
installations = fetch_dataverse_installations()

# Filter for specific installations
harvard = next(i for i in installations if "harvard" in i.name.lower())
demo = next(i for i in installations if "demo" in i.name.lower())

# Create connections
harvard_server = DataverseServer(installation=harvard)
demo_server = DataverseServer(installation=demo)

# Search both servers
harvard_results = harvard_server.search_simple("education")
demo_results = demo_server.search_simple("education")

print(f"Harvard: {harvard_results['data']['total_count']} results")
print(f"Demo: {demo_results['data']['total_count']} results")

Step 5: Retrieve Multi-Standard Dataset Metadata

Retrieve full native JSON metadata or standard export formats (Croissant, DDI, Schema.org, DataCite) using a dataset’s Persistent Identifier (DOI):

pid = "doi:10.5683/SP3/FNS9EF"

# 1. Native Dataverse JSON
dataset = server.get_dataset(pid)
print(f"Title: {dataset['data']['latestVersion']['metadataBlocks']['citation']['fields'][0]['value']}")

# 2. Croissant ML (JSON-LD) for ML pipelines
croissant_data = server.get_dataset_export(pid, exporter="croissant")

# 3. DDI Codebook 2.5 (XML) for variable documentation
ddi_xml = server.get_dataset_export(pid, exporter="ddi")

# 4. Schema.org (JSON-LD) for search indexing
schema_json = server.get_dataset_export(pid, exporter="schema.org")

# 5. DataCite (XML) for citation indexing
datacite_xml = server.get_dataset_export(pid, exporter="datacite")

Step 6: Incremental Harvesting & Statistics

You can use the harvester engine via CLI or Python:

# Query global repository stats and tabular data file counts
dartfx-dataverse stats --country NL

# Incrementally sync multi-standard metadata (Croissant, Native, DDI, Schema.org, DataCite)
dartfx-dataverse harvest ./harvested_records --server dataverse.harvard.edu --format all --limit 10

Or in Python:

from dartfx.dataverse import fetch_server_stats

stats = fetch_server_stats("dataverse.harvard.edu")
print(f"Datasets: {stats['datasets']:,} | Tabular Files: {stats['tabular_files']:,} ({stats['tabular_pct']}%)")

Handling Errors

The package provides structured error handling:

from dartfx.dataverse import DataverseServer, DataverseApiError, ServerInstallation

try:
    server = DataverseServer(
        installation=ServerInstallation(
            name="Test Server",
            hostname="invalid.example.com"
        )
    )
    results = server.get_server_info()
except DataverseApiError as e:
    print(f"API Error: {e.message}")
    print(f"Status Code: {e.status_code}")
    print(f"URL: {e.url}")
except Exception as e:
    print(f"Unexpected error: {e}")

Configuring Caching

The package uses request caching by default. You can configure it:

import requests_cache
from datetime import timedelta

# Create a custom cache session
session = requests_cache.CachedSession(
    cache_name='my_dataverse_cache',
    backend='sqlite',
    expire_after=timedelta(hours=1)
)

# Use with server
server = DataverseServer(
    installation=harvard,
    session=session
)

Disable caching if needed:

import requests

# Use regular requests session (no caching)
server = DataverseServer(
    installation=harvard,
    session=requests.Session()
)

Next Steps

Now that you understand the basics, explore:

Common Use Cases

Finding Datasets by Subject

params = SearchParameters(
    q="subject:medicine",
    type="dataset",
    per_page=50
)
results = server.search(params)

Searching Within a Collection

params = SearchParameters(
    q="*",
    subtree="myDataverse",  # Collection identifier
    type="dataset"
)
results = server.search(params)

Getting Help

If you encounter issues or have questions: