I've got some unruly parquet files generated by so...
# questions-and-troubleshooting
s
I've got some unruly parquet files generated by some legacy python. The arrow schema include some structs which, for some files, are actually null (not null struct values, just null). Duckdb and clickhouse, for example, handle these files and can read all data. I'm trying to work around:
Copy code
Apache Arrow type (nested) null does not match the type STRUCT in StarRocks: BE:10001
Of course, I can modify the files upstream but are there hacks that would allow me to load these files and "fix" anything using starrocks directly? Here is the inferred schema using DESC FILES():
Copy code
Field   Type    Null
title	varchar(1048576)	YES
issue	varchar(1048576)	YES
pages	varchar(1048576)	YES
abstract	varchar(1048576)	YES
journal	varchar(1048576)	YES
authors	array<struct<lastname varchar(1048576), forename varchar(1048576), initials varchar(1048576), identifier varchar(1048576), affiliation varchar(1048576)>>	YES
pubdate	varchar(1048576)	YES
pmid	varchar(1048576)	YES
mesh_terms	varchar(1048576)	YES
publication_types	varchar(1048576)	YES
chemical_list	varchar(1048576)	YES
keywords	varchar(1048576)	YES
doi	varchar(1048576)	YES
references	array<struct<citation varchar(1048576), pmid varchar(1048576)>>	YES
delete	boolean	YES
languages	varchar(1048576)	YES
vernacular_title	varchar(1048576)	YES
pmc	varchar(1048576)	YES
other_id	varchar(1048576)	YES
medline_ta	varchar(1048576)	YES
nlm_unique_id	varchar(1048576)	YES
issn_linking	varchar(1048576)	YES
country	varchar(1048576)	YES
grant_ids	array<struct<grant_id varchar(1048576), grant_acronym varchar(1048576), country varchar(1048576), agency varchar(1048576)>>	YES
_inserted_at	varchar(1048576)	YES
_read_from	varchar(1048576)	YES