Automatic conversion of metadata to np.str_

[JuliaSprenger]

Since b3c21c3 (version 1.5.2) blocks can no longer take names of type np.str_. This causes an issue in neo, where the name of a nix object is stored as a numpy str_ array.

A minimal failing example is

import numpy as np
import nixio as nix
with nix.File('test.nix') as nix_file:
    nix_file.create_block(np.str_('name'), "my_type")

with the error traceback:

    nix_file.create_block(np.str_('name'), "my_type")
  File "nixpy/nixio/file.py", line 425, in create_block
    block = Block.create_new(self, self._data, name, type_, compression)
  File "nixpy/nixio/block.py", line 55, in create_new
    newentity = super(Block, cls).create_new(nixfile, nixparent, h5parent,
  File "nixpy/nixio/entity.py", line 30, in create_new
    h5group.set_attr("name", name)
  File "nixpy/nixio/hdf5/h5group.py", line 253, in set_attr
    self.group.attrs[name] = value
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/_hl/attrs.py", line 103, in __setitem__
    self.create(name, data=value)
  File "h5py/_hl/attrs.py", line 180, in create
    htype = h5t.py_create(original_dtype, logical=True)
  File "h5py/h5t.pyx", line 1663, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1687, in h5py.h5t.py_create
  File "h5py/h5t.pyx", line 1753, in h5py.h5t.py_create
TypeError: No conversion path for dtype: dtype('<U4')

See also NeuralEnsemble/python-neo#1063

[jgrewe]

Hi @JuliaSprenger i am not convinced that this error is related to the changes to the string handling introduced with 1.5.2.
At least in the minimal example, we never go to the h5dataset.read_data method. I wonder whether using np.str_ would have worked at any point in time.
In the minimal example we just pass the "name" to the h5py method to set the attribute of the h5group which apparently does not like the np.str_ dtype.
Is it required for you to support this in neo?

[JuliaSprenger]

Hi @jgrewe Yes, you are right, it seems this was only the tip of the iceberg. The bottom is related to the change of dtypes for metadata. Here is an extended example based on what is happening on the neo side.

I can fix this on the neo side, but maybe it would be worth preventing errors like this also for other applications?

import nixio as nix

import os
for f in ['test.nix', 'test1.nix']:
    if os.path.exists(f):
        os.remove(f)

# first writing block with `str` metadata
with nix.File('test.nix') as nix_file:
    bl = nix_file.create_block('my_nix_name', "my_type")
    bl.metadata = nix_file.create_section('my_nix_name', "neo.block.metadata")
    bl.metadata['nix_custom_name'] = 'my custom block name'

# reading block again, metadata type changed after commit b3c21c3
with nix.File('test.nix', 'r') as nix_file:
    bl = nix_file.blocks[0]
    meta = bl.metadata

    # This is 'str' before and 'np.str_' after commit b3c21c3
    my_custom_name = meta.props[0].values[0]
    print(f'Data type of stored metadata values: {type(my_custom_name)}')

# This change of data dtype causes an error when writing the block with its custom name
with nix.File('test2.nix', 'w') as nix_file:
    nix_file.create_block(my_custom_name, 'my_type')

[jgrewe]

@JuliaSprenger I hope the fix in pull request #536 solves the failing tests in neo, would be glad, if you could try it.

[JuliaSprenger]

Thanks for looking into this @jgrewe! I can confirm that #536 fixes the issue.