Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.4.0] - 2026-10-11

Added

  • Multi-Threaded Harvester Engine (dartfx-nada harvest): * Concurrent worker execution (--workers / -w, default: 4) using ThreadPoolExecutor with thread-safe manifest tracking, error recording, file writes, and Rich progress visualization. * Stabilized download progress bar layout by anchoring server name on the left, fixing bar/count/timer widths, and placing the active filename on the right-hand side to eliminate visual shifting. * Fixed harvest timer display by resetting progress state upon transition from catalog discovery to study synchronization and using TimeElapsedColumn for active stopwatch timing. * Cumulative catalog cache merging and partial-cache expansion handling, ensuring higher --limit requests query live servers while retaining previously cached studies. * Comprehensive harvest error persistence in .manifest.json ensuring both transient (recoverable) and permanent failures are captured, queryable via dartfx-nada errors, and automatically retried on subsequent runs. * Robust parsing of string-formatted total study counts returned by some NADA server APIs (e.g. instances like microdata.ubos.org:7070), preventing Rich progress bar crashes and infinite search pagination loops.

  • Expanded Metadata Standards: * Added support for MLCommons Croissant (croissant) and Google Dataset Search Schema.org (schema / .schema.jsonld) in dartfx-nada study and metadata harvester subsystem (--formats schema). * New export methods on NadaServer: get_study_croissant() and get_study_schema().

  • Server Discovery & Registry Improvements: * Added list command as the preferred command for discovering NADA servers, with installations retained as a backwards-compatible alias. * Active clickable hyperlinks and CSV/JSON output formatting in dartfx-nada list.

  • Diagnostic & Resilience Enhancements: * Abbreviated status error badges (e.g. SSL Hostname Mismatch, Connection Failure, Timeout (10s)) replacing verbose connection pool stack traces in dartfx-nada stats. * Domain parking / domain-for-sale detection and strict JSON/CSV payload validation in server stats probing. * Streamlined stats table layout by removing the redundant Collections column in favor of Studies, Latency, and Status. * Fixed study count extraction for NADA v4 servers (e.g., nada.stat.cm, microdata.statistics.gov.rw) by parsing catalog search count containers in catalog HTML. * Bidirectional HTTP/HTTPS protocol resilience: Automatic fallback to HTTP on SSL verification failures with recovered study counts (HTTP (SSL Failed)), and HTTPS availability probes for legacy HTTP servers with upgrade recommendations (Online (HTTPS Avail)). * Enriched dartfx-nada info command displaying organization name, catalog URL, detected version, live/cached total study count, collections count, protocol health advisory, and a comprehensive Supported Metadata & Export Formats matrix. * Comprehensive Server Status & Diagnostics documentation in IMPLEMENTATION.md, README.md, and Sphinx docs (docs/source/diagnostics.md).

[0.2.0] - 2026-10-10

Added

  • First-Class Metadata Harvester & Sync Subsystem (dartfx-nada harvest): * Incremental, hash-verified synchronization of study metadata across NADA servers. * Multi-format harvesting supporting DDI 2.5 CodeBook XML (ddi), Study Metadata JSON (json), and External Documentation/Resources (resources), as well as --formats all. * SHA-256 integrity verification and fast timestamp checking backed by .manifest.json. * Deletion pruning protection during filtered sync runs (only prune deletions during full, unconstrained server syncs). * 24-hour local catalog caching (.catalog_cache.json) and server statistics caching (.stats_cache.json) with --refresh (-r) options. * Granular error classification engine (classify_harvest_error, analyze_harvest_errors, render_harvest_errors) categorizing Not Found, Authentication Required, Restricted Study, WAF/Bot Protection, Server Errors, Timeouts, and SSL/TLS diagnostics. * Non-recoverable error caching to prevent redundant requests on consecutive harvest runs. * Automated API token discovery and persistence via .api_token, .nada_tokens.json, and environment variables (NADA_API_TOKEN / NADA_{HOST}_API_TOKEN).

  • Server Discovery & Registry Subsystem: * Centralized ServerInstallation Pydantic model with ISO 3166-1 Alpha-2 country_code derivation, clean_hostname, and url helper properties. * Comprehensive ISO 3166-1 country crosswalk (COUNTRY_TO_ISO2, get_iso2_code, matches_country). * fetch_nada_installations registry engine with remote/local fallback and dynamic synthesis for unlisted/private servers. * Active clickable hyperlinks and CSV/JSON output formatting in dartfx-nada installations.

  • Core Server Client (`NadaServer`): * Full Pydantic v2 model inheriting from BaseModel. * Automatic version detection (detect_server_version) dynamically handling NADA v5 (/api/) and NADA v4 (/index.php/api/) endpoint route prefixes. * Integrated requests-cache session management. * Study export methods (get_study_ddi, get_study_json, get_study_resources). * Search query validation via SearchParameters Pydantic model.

  • Server Statistics & Diagnostics Subsystem (dartfx-nada stats): * Multi-threaded concurrent query fetching via ThreadPoolExecutor for low response latency. * Semantic color-coded performance tier badges (🟢⚡ <3s Fast, 🟢 3–8s Normal, 🟡 8–15s Moderate, 🔴 >15s Degraded, 🔒 Protected, ❌ Failed). * Deep SSL/TLS diagnostics isolating missing intermediate CA chains, expired certificates, hostname mismatches, and self-signed certificates.

  • Modern CLI & Tooling Architecture: * Full-featured CLI powered by Typer and Rich with smart dash-to-underscore option normalization. * CLI commands: installations, info, collections, search, study, harvest, stats, errors. * Comprehensive unit test suite covering core modules, CLI commands, and harvester workflows.

Removed

  • Removed deprecated legacy NadaCatalog, NadaStudy, and NadaHarvester classes/aliases, establishing NadaServer and ServerHarvester as the streamlined canonical interfaces.

[0.1.0] - 2024-03-01

Added

  • Initial release of the NADA toolkit.

  • Basic NadaServer, NadaCatalog, and NadaStudy classes.

  • Basic search and study metadata retrieval.