I think it's a bug that we didn't carry the

OpWrite

in

gcs_initiate_multipart_upload

We should set them while starting the upload process.

It's worth to add a test to set content-type for a large file, like 16MiB

That makes sense, another file that is 27.6 MiB large does not have headers either.. I just didn't notice it at first, because it was supposed to be private anyway

Maybe a workable stopgap solution would be to upload the whole file in one... only, I am afraid to compress whole files instead of streaming compression

Or is there an actual good and quick fix here?

You can try op.write_with(path).chunk(128MiB).await;

This can ensure that all files under 128MiB will upload in oneshot.

I meant a bug fix, not a workaround

o, bug fix is easy.

Like what we do in

gcs_insert_object_request

, add

OpWrite

for this function and set the correct header.

Okay, thanks. Will probably try that next week

Thank you in advance!

I submitted a PR. I felt lazy, so I packaged a fix for another breaking but small-scale bug. I hope you don't mind https://github.com/apache/opendal/pull/6275

Great work!

I tested this on our project, and the headers now work on multipart upload with these changes, as does the presign feature

That's nice.

My PR also adds custom metadata in multipart upload, but I didn't test that part since I have never used them

hi, I am trying to use the http operator in the python bindings to access a url with a custom certificate, but i get:

error sending request for url (https://internal/index.html): client error (Connect): invalid peer certificate: UnknownIssuer

with:

python3 -c "from opendal import Operator; print(Operator('http', endpoint='https://internal').read('index.html').decode().splitlines()[0])"

I have added the certificate to the system, and it works with curl, but I can't get it to work. It seems to be related to the features enabled by reqwest, but I guess I can't control from the python library? Is there any way around this?

Your certificate is not going to be part of WebPKI the HTTP client reqwest uses. I don't think you can add a certificate to the compiled certs. There are many tunnel solutions over internet you might want to try.

sure but rustls-native-certs used to be enabled for reqwest https://github.com/apache/opendal/pull/4365/files#diff-ac4e3071598a47848866c532150a233af5159d2757dab53572781ed09e13fbeeL66 so while one can enable it using rust, I think it would be useful for those using bindings to easily use system certs

I agree with you. The core is working on with external http client that will fix this problem. And then adding a feature flag is straightforward for bindings. This is a resource problem. For now though, yes, you are free to build from code.

Can someone explain what the root argument is used for? Is this where temporary operations are performed in the event of reads/writes? It seems odd to have to do the following... (I am attempting to implement a Pathlib like interface over OpenDAL I intend to propose to pyiceberg in favor of fsspec and pyarrow file systems)```py from urllib.parse import urlparse, ParseResult from opendal import Operator from pathlib import Path class OperatorHelper: def __init__(self, operator: Operator, parsed: ParseResult, path: Path) -> None: self.operator = operator self.parsed = parsed self.path = path class FileSystem: @staticmethod def _get_operator_helper(path: str, **options) -> OperatorHelper: """Get the OperatorHelper for the given path. Args: path: The path to the file or directory. Returns: OperatorHelper """ parsed = urlparse(path) scheme = parsed.scheme netloc = parsed.netloc path = Path(parsed.path) root = str(path.parent) if scheme in ("fs", ""): operator = Operator("fs", root=root, **options) elif scheme in ("s3", "gcs"): operator = Operator(scheme, root=root, bucket=netloc) return OperatorHelper(operator, parsed, path) def read(self, path: str, **options): helper = self._get_operator_helper(path, **options) name = helper.path.name return helper.operator.read(name) file_system = FileSystem() ```

And if that is the case, how am I supposed to read a file that doesnt exist in the root? Operators are truly supposed to be so short lived that I should dispose of one and create a new one for every operation? For example if I was writing a file larger than memory from local to s3 I would need several operators right?

is there a way to set a unknow size in stat etc? or should i just make up a number (i dont know the size)

@Xuanwo Hey hi This is Aksshay I am really happy to be here and I am new to rust and to openDAL I want to contribute to this openDAL where should I start with can you pls help me !

Hi, I'm curious what is the proper way of using HttpClientLayer with a custom client that impls HttpFetch? It seems the HttpClientLayer::new() takes in only an HttpClient?

I saw magnus getting new release https://github.com/matsadler/magnus/releases/tag/0.8.0 I could start work on Ruby binding 😄 (the time is tricky...)