Harvester Module

The harvester module provides classes and functions for incremental multi-standard metadata synchronization, global repository statistics inspection, error classification, and token management.

ServerHarvester

class dartfx.dataverse.ServerHarvester(host, repo_root)

Bases: object

Manages harvesting, diff detection, and file persistence for a single Dataverse server.

__init__(host, repo_root)

sync(query=None, since_date=None, dry_run=False, limit=None, metadata_formats='croissant', verbose=False, target_doi=None, refresh_catalog=False, cache_ttl_hours=24.0, retry_errors=False, force_verify=False, tabular_only=True, api_token=None, progress_callback=None)

Perform intelligent incremental sync (Additions, Updates, Deletions).

Functions

Dataset Catalog & Fetching

dartfx.dataverse.fetch_active_datasets(host, query=None, since_date=None, limit=None, target_doi=None, server_dir=None, refresh_catalog=False, cache_ttl_hours=24.0, per_page=100, verbose=False, tabular_only=True, api_token=None)

Search for active datasets using Search API or DOI target with local 24h catalog caching.

dartfx.dataverse.harvester.fetch_metadata_record(host, pid, metadata_format='croissant', verbose=False, api_token=None)

Fetch dataset metadata in the specified format. Returns (content_bytes, file_extension, error_reason).

dartfx.dataverse.harvester.fetch_oai_deletions(host, since_date=None, verbose=False, api_token=None)

Query Dataverse OAI-PMH endpoint to find deleted dataset PIDs.

Repository Statistics & Profiling

dartfx.dataverse.fetch_server_stats(host, query=None, api_token=None, timeout=15, repo_root=None, refresh_cache=False, cache_ttl_hours=24.0)

Retrieve counts of datasets, total files, and tabular data files from a Dataverse server.

dartfx.dataverse.format_version(version)

Format a Dataverse version string into a clean semantic version (e.g. ‘v1.3.1’, ‘v5.14’).

Strips git hashes, commit IDs, build metadata, or extra trailing tokens to prevent widening UI columns.

Error Classification & Diagnostics

dartfx.dataverse.classify_harvest_error(reason)

Categorize an error message string into a standardized, descriptive error category.

dartfx.dataverse.analyze_harvest_errors(repo_dir=None, server=None)

Scan repository storage directories for .manifest.json files and aggregate error statistics.

Parameters:

• repo_dir (Path | str | None) – Path to the harvested repository root directory or a specific server directory. If None, checks DARTFX_DATAVERSE_REPOSITORY env var or current directory.

• server (str | None) – Optional server hostname filter (or ‘ALL’).

Returns:

• total_errors: Total number of failed record attempts

• total_datasets: Number of distinct datasets affected

• servers_with_errors: Number of server repositories with recorded errors

• by_type: dict mapping error type to count

• by_format: dict mapping metadata format to count

• by_server: dict mapping server hostname to count

• matrix: dict mapping error type to {format: count}

• records: list of individual error dicts with details

Return type:

Structured dictionary containing

dartfx.dataverse.harvester.render_harvest_errors(analysis, by_format=False, by_server=False, details=False, console_out=None)

Render Rich visual reports of harvest errors.

Token & Configuration Management

dartfx.dataverse.resolve_server_token(host=None, repo_root=None, explicit_token=None)

Resolve API token for a Dataverse server by searching in order: 1. Explicitly supplied CLI token (–api-token / -k) 2. Server-specific directory token file (<repo_root>/<host>/.api_token or ./<host>/.api_token) 3. Central repository root tokens mapping (<repo_root>/.dataverse_tokens.json) 4. Current working directory tokens (.dataverse_tokens.json or .tokens.json) 5. Server-specific environment variable (e.g. DATAVERSE_API_TOKEN_DATAVERSE_UNC_EDU) 6. Global environment variable (DATAVERSE_API_TOKEN or DATAVERSE_KEY)

dartfx.dataverse.save_server_token(host, token, repo_root=None)

Save an API token for a server to both its server directory (.api_token) and central .dataverse_tokens.json.

Utility Functions

dartfx.dataverse.harvester.sanitize_pid(identifier)

Convert DOI or Handle PID to safe directory name.

dartfx.dataverse.harvester.get_format_extension(fmt)

Return static file extension for a given metadata format.

dartfx.dataverse.harvester.normalize_formats(formats_input)

Parse comma-separated or list of metadata formats into a clean list.