A few bugs to squash to get dinf running on Python 3.13/3.14

[YourePrettyGood]

Hi,
Thanks for all your work in developing dinf! It's been interesting to see how ML models get adapted to demographic inference, and both pg-gan and dinf seem like big steps forward, especially incorporating posterior estimates.

Over the past week, I've been trying to get dinf working with a simple conda build, and ran into a few bugs that needed squashing to run the example models. In general, I've tested these from a conda install of this form (with conda-forge and bioconda already as default channels):

conda create -n dinf-gpu dinf bcftools "python<3.14"

This is pulling the following packages, amongst others:

bioconda/linux-64::bcftools-1.23.1-hb2cee57_0
bioconda/noarch::dinf-0.5.0-pyhdfd78af_0
bioconda/linux-64::htslib-1.23.1-h633afcb_0
conda-forge/noarch::jax-0.10.2-pyhd8ed1ab_0 
conda-forge/linux-64::jaxlib-0.10.1-cuda129_py313h1656396_200
conda-forge/linux-64::numpy-2.5.0-py313hf6604e3_0 
conda-forge/linux-64::python-3.13.14-h6add32d_100_cp313

With that installation, the simulation-only examples work perfectly fine. The real data examples, on the other hand, require the first two bug fixes.

Bug 1: 1000 Genomes metadata parsing

The first one was the biggest pain to troubleshoot, as the exception was thrown in a totally different section of the code. Using NumPy 2.0+, np.recfromtxt() has been removed from the main NumPy namespace, so this line:

dinf/dinf/vcf.py

Line 27 in 6a206ca

data = np.recfromtxt(filename, names=True, encoding="ascii")

needs to be replaced with:

    data = np.genfromtxt(filename, names=True, dtype=['<U16', '<U16', '<U16', '<U16', 'i2', '<U16', '<U16'], encoding="ascii").view(np.recarray)

The first layer to that one is the obvious initial exception thrown about np.recfromtxt not existing. But when I replaced np.recfromtxt() with np.genfromtxt().view(np.recarray) without setting the dtype explicitly, dtype defaults to a float8, which then parses all the sample IDs and parental IDs as nans, which then produces empty self.samples in BagOfVcf, and cascades down to an AttributeError: 'NoneType' object has no attribute 'array' from this line:

dinf/dinf/vcf.py

Line 131 in 6a206ca

a = variant.genotype.array()

Specifying dtype=None also doesn't work, at least for 20130606_g1k_3202_samples_ped_population.txt, as there are a whole series of exceptions thrown for parsing columns into the wrong type. So the auto-detection of column types doesn't seem to work properly, at least for this file and invocation. I just threw together that explicit dtype list, but I'm pretty bad with dtype selection, so feel free to tweak it however you want.

Bug 2: dinf plot gan assumes truth values for parameters are set in model

Any of the example models that use real data generally don't specify a truth value when specifying the parameter ranges, yet the plotting code assumes one is present here:

dinf/dinf/plot.py

Lines 1192 to 1194 in 6a206ca

	if parameters is not None and x_param != "_Pr":
	truth = parameters[x_param].truth
	ax.axhline(truth, c="red", ls="-", alpha=0.7)

So it's probably worth adding this to the conditional:

                if parameters is not None and x_param != "_Pr" and parameters[x_param].truth is not None:
                    truth = parameters[x_param].truth
                    ax.axhline(truth, c="red", ls="-", alpha=0.7)

otherwise you get an exception like:

...
  File "/home/pfr8/.conda/envs/dinf-gpu/lib/python3.13/site-packages/dinf/plot.py", line 1194, in __call__
    ax.axhline(truth, c="red", ls="-", alpha=0.7)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pfr8/.conda/envs/dinf-gpu/lib/python3.13/site-packages/matplotlib/axes/_axes.py", line 820, in axhline
    scaley = (yy < ymin) or (yy > ymax)
   
TypeError: '>' not supported between instances of 'float' and 'NoneType'

Bug 3: Python 3.14 Union dunder attributes are now read-only

This one's a bit silly, but these lines need to be commented out or reworked for compatibility with Python 3.14:

dinf/dinf/dinf_model.py

Lines 17 to 19 in 6a206ca

	FeatureCollection.__doc__ = (
	"""A feature array or a labelled collection of feature arrays."""
	)

See here for reference:
python/cpython#141829

Other minor suggestions

• There's an off-by-one error in the example model.py files for any examples that use real data, since range() is right-exclusive, so range(21) should be range(22) here, for example:
  

  dinf/examples/post_ooa/model.py

  Line 19 in 6a206ca

  keep_contigs={f"chr{c + 1}" for c in range(21)}, # Exclude chrX, etc.

• Consider including bcftools in the suggested conda environment and Docker container in case the end user needs to convert their VCFs to BCF or wants to troubleshoot their inputs
• Documentation and usage still references dinf.plot and dinf.tabulate rather than dinf-plot and dinf-tabulate, presumably due to these two lines:
  

  dinf/dinf/plot.py

  Line 1203 in 6a206ca

  prog="dinf.plot", description="Dinf plotting tools."

  
  

  dinf/dinf/tabulate.py

  Line 220 in 6a206ca

  prog="dinf.tabulate", description="Tabulate Dinf output."

And a final note/curiosity:
It appears that neither BagOfVcf._close() nor BagOfVcf._reopen() is ever actually triggered, as you're using spawn rather than fork for your multiprocessing.Pool context, so the os.register_at_fork() call doesn't actually do anything (at least on our Linux cluster). As far as I can tell, everything works fine despite this, whether with -j 1 or -j 8.

All in all, thanks for developing dinf and providing such helpful documentation!

Best regards,
Patrick Reilly

[YourePrettyGood]

Sorry, I also forgot to mention another tiny bug in a couple of the example model.pys:
Any model.py that calls dinf.get_samples_from_1kgp_metadata() needs to specify populations=populations as a named argument. The model.py for examples/gutenkunst2009/ is fine, but those for examples/one_population_growth/, examples/post_ooa/, and examples/two_pop_ooa/ are not.

[grahamgower]

Hi @YourePrettyGood. Thanks for taking the time to provide such detailed feedback and bug reports. You may have noticed that this repository hasn't seen any attention recently, and it seems that Python projects suffer bitrot really quickly!

I'm no longer working in popgen, nor in academic research, so it's unlikely that I'll commit time to maintaining dinf. Its a bit of a shame, because there are a whole bunch of improvements and additions that I had thought about. As you've seen, the code for working with empirical data is in particular need of improvement.

[grahamgower]

And a final note/curiosity:
It appears that neither BagOfVcf._close() nor BagOfVcf._reopen() is ever actually triggered, as you're using spawn rather than fork for your multiprocessing.Pool context, so the os.register_at_fork() call doesn't actually do anything (at least on our Linux cluster). As far as I can tell, everything works fine despite this, whether with -j 1 or -j 8.

I think I was previously using fork(), but then I left it in after changing to spawn() to hedge my bets about using dinf as library and not just from the cli. I think there are still tests using fork? But, yeah, this is probably dead code.

[YourePrettyGood]

Hi Graham,
Thanks for responding! Yeah, given the lack of activity for a couple years, I wasn't sure anyone was going to respond, but figured an Issue might help anyone else who runs into this.

That's unfortunate for the rest of us in Pop Gen, but totally understandable. Do you know if anybody from Fernando's lab might take up the mantle?

If it's okay with you, I'll fork and might send a PR or two. But if you'd prefer not to get the spam, I'll avoid the PRs. Just let me know.

Best regards,
Patrick Reilly

[grahamgower]

Please open PRs. I'd be happy to see the activity here! Sadly, there wasn't anyone else in the lab working on this except very early on with the project. (It started as a Master project, before I shifted the focus and rewrote it all).