unihan-etl ·

An ETL tool for the Unicode Han Unification (UNIHAN) database releases: unihan-etl downloads, unpacks, and converts Unicode's raw CJK character data files into a flattened, tabular export or a structured, hierarchical one.

unihan-etl serves dual purposes: a typed Python library offering an API for working with the data as Python objects, and a CLI for exporting to CSV, JSON, or YAML.

unihan-etl is a component of the cihai suite of CJK related projects. For a similar tool, see libUnihan.

As of v0.31.0, unihan-etl is compatible with UNIHAN Version 15.1.0 (released on 2023-09-01, revision 35).

The UNIHAN database

The UNIHAN database spreads its data across multiple files. A sample:

U+3400	kCantonese		jau1
U+3400	kDefinition		(same as U+4E18 丘) hillock or mound
U+3400	kMandarin		qiū
U+3401	kCantonese		tim2
U+3401	kDefinition		to lick; to taste, a mat, bamboo bark
U+3401	kHanyuPinyin		10019.020:tiàn
U+3401	kMandarin		tiàn

Field values vary in shape by field type. kHanyuPinyin maps a codepoint to an entry in Hànyǔ Dà Zìdiǎn; 10019.020:tiàn is one such entry. Some fields hold several:

U+5EFE	kHanyuPinyin		10513.110,10514.010,10514.020:gǒng
U+5364	kHanyuPinyin		10093.130:xī,lǔ 74609.020:lǔ,xī

kHanyuPinyin delimits multiple entries with spaces, separates a location from its pinyin reading with :, and separates multiple locations or readings with ,. This is one of 90 fields in the database.

Tabular, "Flat" output

CSV (default)

$ unihan-etl

char,ucn,kCantonese,kDefinition,kHanyuPinyin,kMandarin
㐀,U+3400,jau1,(same as U+4E18 丘) hillock or mound,,qiū
㐁,U+3401,tim2,"to lick; to taste, a mat, bamboo bark",10019.020:tiàn,tiàn

With $ unihan-etl -F yaml --no-expand:

- char: 㐀
  kCantonese: jau1
  kDefinition: (same as U+4E18 丘) hillock or mound
  kHanyuPinyin: null
  kMandarin: qiū
  ucn: U+3400
- char: 㐁
  kCantonese: tim2
  kDefinition: to lick; to taste, a mat, bamboo bark
  kHanyuPinyin: 10019.020:tiàn
  kMandarin: tiàn
  ucn: U+3401

To preview in the CLI, try tabview or csvlens.

JSON

$ unihan-etl -F json --no-expand

[
  {
    "char": "㐀",
    "ucn": "U+3400",
    "kDefinition": "(same as U+4E18 丘) hillock or mound",
    "kCantonese": "jau1",
    "kHanyuPinyin": null,
    "kMandarin": "qiū"
  },
  {
    "char": "㐁",
    "ucn": "U+3401",
    "kDefinition": "to lick; to taste, a mat, bamboo bark",
    "kCantonese": "tim2",
    "kHanyuPinyin": "10019.020:tiàn",
    "kMandarin": "tiàn"
  }
]

Tools:

• View in CLI: python-fx, jless or fx.
• Filter via CLI: jq, jql, gojq.

YAML

$ unihan-etl -F yaml --no-expand

- char: 㐀
  kCantonese: jau1
  kDefinition: (same as U+4E18 丘) hillock or mound
  kHanyuPinyin: null
  kMandarin: qiū
  ucn: U+3400
- char: 㐁
  kCantonese: tim2
  kDefinition: to lick; to taste, a mat, bamboo bark
  kHanyuPinyin: 10019.020:tiàn
  kMandarin: tiàn
  ucn: U+3401

Filter via the CLI with yq.

"Structured" output

A codepoint can carry more structure than a flat row holds. unihan-etl extracts it uniformly and prunes empty values, exporting to JSON, YAML, and Python list/dicts.

Why not CSV?

CSV only holds table-like data. JSON and YAML also hold key-values and hierarchical entries.

JSON

$ unihan-etl -F json

[
  {
    "char": "㐀",
    "ucn": "U+3400",
    "kDefinition": ["(same as U+4E18 丘) hillock or mound"],
    "kCantonese": ["jau1"],
    "kMandarin": {
      "zh-Hans": "qiū",
      "zh-Hant": "qiū"
    }
  },
  {
    "char": "㐁",
    "ucn": "U+3401",
    "kDefinition": ["to lick", "to taste, a mat, bamboo bark"],
    "kCantonese": ["tim2"],
    "kHanyuPinyin": [
      {
        "locations": [
          {
            "volume": 1,
            "page": 19,
            "character": 2,
            "virtual": 0
          }
        ],
        "readings": ["tiàn"]
      }
    ],
    "kMandarin": {
      "zh-Hans": "tiàn",
      "zh-Hant": "tiàn"
    }
  }
]

YAML

$ unihan-etl -F yaml

- char: 㐀
  kCantonese:
    - jau1
  kDefinition:
    - (same as U+4E18 丘) hillock or mound
  kMandarin:
    zh-Hans: qiū
    zh-Hant: qiū
  ucn: U+3400
- char: 㐁
  kCantonese:
    - tim2
  kDefinition:
    - to lick
    - to taste, a mat, bamboo bark
  kHanyuPinyin:
    - locations:
        - character: 2
          page: 19
          virtual: 0
          volume: 1
      readings:
        - tiàn
  kMandarin:
    zh-Hans: tiàn
    zh-Hant: tiàn
  ucn: U+3401

Features

• downloads and caches UNIHAN releases from unicode.org
• follows the field semantics in UNIHAN's database design
• exports to CSV, JSON, and YAML (YAML requires pyyaml) via -F
• exports a subset of fields via -f
• expands multi-value delimited fields into structured lists/dicts for YAML, JSON, and Python exports
• handles the Unicode-heavy encoding of UNIHAN's source files
• data package descriptor included
• core component and dependency of cihai, a CJK library
• Python 3.10+ and PyPy

If you encounter a problem or have a question, please create an issue.

Installation

Using uv to add the CLI to your project:

$ uv add unihan-etl

Using pip:

$ pip install --user unihan-etl

Run the tool without a persistent install via uvx:

$ uvx unihan-etl

or by pipx:

$ pipx install unihan-etl

Developmental releases

Using uv, opt-in to pre-release versions:

$ uv add --prerelease=allow unihan-etl

To pin a specific pre-release (for example 0.27.0a1):

$ uv add --prerelease=allow 'unihan-etl==0.27.0a1'

pip:

$ pip install --user --upgrade --pre unihan-etl

pipx:

$ pipx install --suffix=@next 'unihan-etl' --pip-args '\--pre' --force

This installs the pre-release under the unihan-etl@next executable name, alongside a regular install. Run it the same way, for example unihan-etl@next export.

Run pre-release builds without installing with uvx:

$ uvx --prerelease=allow unihan-etl

Or pinned to that example version:

$ uvx --from 'unihan-etl==0.27.0a1' unihan-etl

Swap 0.27.0a1 for whichever pre-release you plan to use.

Usage

Without --destination, an export writes to the platform's XDG data directory (see Code layout) and the download is cached, so a repeat run skips the network. A failed export exits non-zero and writes both a traceback and a one-line Error: <message> to stderr. See unihan-etl CLI arguments for every flag, including how to specify columns, files, and download URLs.

To output CSV, the default format:

$ unihan-etl

To output JSON:

$ unihan-etl -F json

To output YAML:

Add PyYAML with uv:

$ uv add pyyaml

Or install it with pip:

$ pip install --user pyyaml

Then run:

$ unihan-etl -F yaml

To only output the kDefinition field in a csv:

$ unihan-etl -f kDefinition

To output multiple fields, separate with spaces:

$ unihan-etl -f kCantonese kDefinition

To output to a custom file:

$ unihan-etl --destination ./exported.csv

To output to a custom file (templated file extension):

$ unihan-etl --destination ./exported.{ext}

See unihan-etl CLI arguments for advanced usage examples.

Code layout

# cache dir (Unihan.zip is downloaded, contents extracted)
{XDG cache dir}/unihan_etl/

# output dir
{XDG data dir}/unihan_etl/
  unihan.json
  unihan.csv
  unihan.yaml   # (requires pyyaml)

# package dir
unihan_etl/
  core.py    # argparse, download, extract, transform UNIHAN's data
  options.py    # configuration object
  constants.py  # immutable data vars (field to filename mappings, etc)
  expansion.py  # extracting details baked inside of fields
  types.py      # type annotations
  util.py       # utility / helper functions

# test suite
tests/*

API

unihan-etl is a typed Python library underneath the CLI; see the full API.

>>> from unihan_etl.core import Packager
>>> pkgr = Packager()
>>> hasattr(pkgr.options, 'destination')
True

Developing

$ git clone https://github.com/cihai/unihan-etl.git

$ cd unihan-etl

See CONTRIBUTING.md for this repository's setup and gates. Bootstrap your environment and learn more about contributing to any cihai project: the suite shares pytest, sphinx, mypy, ruff, tmuxp, and file watcher helpers (e.g. entr(1)) across repositories.

More information