RomajiASR

RomajiASR is a Japanese phoneme ASR project focused on singing transcription.

The project is built on top of prj-beatrice/japanese-hubert-base-phoneme-ctc and extends that line of work with additional training on tens of hours of singing audio. The target output is romanized phoneme tokens (romaji-style phoneme sequences). In internal experiments, this repository achieved better singing transcription accuracy and robustness than the original baseline.

The public source tree is intentionally lightweight:

• training code is included
• PyTorch inference code is included
• ONNX export and ONNX Runtime inference are included
• forced-alignment-related modules are still present, but are experimental
• datasets, manifests, and large training artifacts are not included in the repository

What This Repository Provides

• A HuBERT-based phoneme CTC training pipeline for Japanese singing ASR
• PyTorch inference for checkpoint directories
• ONNX export utilities for deployment or runtime experiments
• ONNX Runtime inference with CPU, CUDA, or DirectML provider selection
• Experimental FA-related modules for alignment-oriented research

Important Notes

• This repository does not ship datasets or training manifests.
• Existing configs may contain local placeholder paths that you should edit for your own environment.
• The released pretrained models are expected to be provided separately in safetensors checkpoint form and ONNX form.
• The currently provided pretrained models do not include FA capability unless explicitly stated otherwise.
• FA-related code should be treated as experimental research code rather than a stable public interface.

Repository Layout

.
|-- configs/            Training and experiment configs
|-- src/                Training, inference, export, evaluation, and FA code
|-- third_party/k2/     Vendored k2 code used by some experimental components
|-- requirements.txt    Base Python dependencies
`-- .gitignore

Main entry points:

• src/train.py: main HuBERT-based training pipeline
• src/infer.py: PyTorch checkpoint inference
• src/export_onnx.py: export a checkpoint to ONNX
• src/infer_onnx.py: run ONNX Runtime inference
• src/train_alignment.py: experimental alignment losses and FA-related training helpers

Installation

Base environment:

pip install -r requirements.txt

For ONNX export and ONNX inference, install additional packages as needed:

pip install onnx onnxruntime

If you want GPU or DirectML execution, use the runtime package that matches your environment, for example:

pip install onnxruntime-gpu

or

pip install onnxruntime-directml

Data Format

This repository expects manifest-style JSONL files prepared by the user.

A minimal phoneme-ASR manifest entry looks like:

{"audio": "path/to/audio.wav", "phones": ["a", "i", "u", "e", "o"]}

Common optional fields used by the training code include:

• duration_sec
• phones_base
• alt_phones
• n_phones
• alignment-related fields for experimental FA / phone-alignment branches

The vocabulary file is expected to be a JSON mapping from token string to integer ID, for example:

{
  "<blank>": 0,
  "<unk>": 1,
  "a": 2,
  "i": 3
}

Training

The main training entry point is:

python src/train.py --config configs/exp8_base_retrain.yaml

To resume from the last saved training state:

python src/train.py --config configs/exp8_base_retrain.yaml --resume_from_checkpoint path/to/last_state.pt

Before running training, make sure you update the config file to point to:

• your pretrained backbone or checkpoint
• your train manifest
• your dev manifest
• your vocabulary JSON
• your output directory

The default configs in configs/ are examples from local experiments, not portable turnkey presets.

PyTorch Inference

Run inference from a checkpoint directory:

python src/infer.py --ckpt path/to/checkpoint_dir --audio path/to/test.wav

Run inference on a manifest:

python src/infer.py --ckpt path/to/checkpoint_dir --manifest path/to/test.jsonl

Optional decoding controls:

• --decode greedy|beam
• --beam_width
• --nbest
• --token_prune_size
• --insertion_penalty
• --length_bonus
• --phoneme_lm
• --lm_weight

If --vocab is omitted, the script tries to auto-detect phoneme_vocab.json from the checkpoint directory.

ONNX Export

Export a checkpoint to ONNX:

python src/export_onnx.py --ckpt path/to/checkpoint_dir --out path/to/model.onnx --opset 17

Example with dynamic time axis:

python src/export_onnx.py --ckpt path/to/checkpoint_dir --dynamic_time --opset 17

Useful export options:

• --export_mode full_logits|minimal_logits|minimal_pred_ids
• --dynamic_time
• --dynamic_batch
• --fp16
• --fp16_io
• --skip_verify

The exporter also writes metadata next to the ONNX file.

ONNX Runtime Inference

Run ONNX inference on a single file:

python src/infer_onnx.py --model path/to/model.onnx --audio path/to/test.wav --provider auto

Run ONNX inference on a manifest:

python src/infer_onnx.py --model path/to/model.onnx --manifest path/to/test.jsonl --provider auto

Available providers:

• auto
• cpu
• cuda
• dml

The ONNX inference path also supports beam search and optional phoneme LM rescoring when the exported model returns logits rather than pre-argmax IDs.

Pretrained Models / Download

Pretrained models are intended to be distributed separately from the source repository, for example through GitHub Releases.

Recommended release contents:

• safetensors checkpoint package
  • config.json
  • model.safetensors
  • phoneme_vocab.json
  • optional: token_expansion_map.json
  • optional: train_config.yaml
• ONNX package
  • model.onnx
  • model.meta.json
  • phoneme_vocab.json

Recommended local layout after download:

checkpoints/
`-- your_model_name/
    `-- best/
        |-- config.json
        |-- model.safetensors
        |-- phoneme_vocab.json
        |-- token_expansion_map.json
        `-- train_config.yaml

PyTorch inference example:

python src/infer.py --ckpt checkpoints/your_model_name/best --audio path/to/test.wav

ONNX inference example:

python src/infer_onnx.py --model path/to/model.onnx --audio path/to/test.wav --provider auto

For public distribution, model.safetensors is preferred over pytorch_model.bin because it is a tensor-only format and avoids pickle-based deserialization.

Experimental FA Support

This repository still contains FA-related modules such as:

• src/fa_modules.py
• src/train_alignment.py
• src/infer_fa_textgrid.py
• src/hfa_textgrid.py

These components are experimental.

Two important limitations:

• the public pretrained models for this repository should not be assumed to support FA
• FA behavior, labels, and alignment supervision formats are more fragile than the main phoneme-ASR path

If your goal is standard singing phoneme transcription, start with the main train.py, infer.py, export_onnx.py, and infer_onnx.py workflow.

Model Scope

This repository is primarily for:

• Japanese singing phoneme transcription
• romaji-style phoneme-token output
• HuBERT-based CTC training and inference
• ONNX export and runtime experiments

It is not intended to be a dataset release repository.

Acknowledgment

This project is based on and inspired by:

• prj-beatrice/japanese-hubert-base-phoneme-ctc

That upstream model line provided the starting point for the singing-focused training and experimentation in this repository.