marsupialtail
04/10/2024, 10:54 PMpatrick_00164
04/12/2024, 5:25 AMwrtier.write
and writer.close
? IIUC, they are not after I went throught the codes.
2. What is the receommeneded way to retry writer API used by writer? I saw that when MultipartWrtier
sees an error, it will recreate the WritePartFuture
. Is it for retry? How can the caller / application drive the retry? Should the caller just ignore errors returned by writer.write
and keep writing?.beepboop666
04/13/2024, 1:08 AMXuanwo
04/15/2024, 3:53 AMXuanwo
04/15/2024, 4:22 AMLink
04/15/2024, 12:10 PMmarsupialtail
04/15/2024, 8:57 PMmarsupialtail
04/15/2024, 10:11 PMvertexclique
04/16/2024, 2:47 PMjohnbampton
04/22/2024, 4:50 PMXuanwo
04/23/2024, 9:47 AMXuanwo
04/23/2024, 2:23 PMrust
async fn test(op: Operator) -> io::Result<()> {
let mut r = op
.reader_with("hello.txt")
.concurrent(8)
.chunk(256)
.await?
.into_futures_async_read(1024..2048);
let mut bs = Vec::new();
r.read_to_end(&mut bs).await?;
Ok(())
}
sreev
04/23/2024, 7:08 PMXuanwo
04/26/2024, 10:24 AMLink
04/26/2024, 10:31 AMXuanwo
04/30/2024, 1:09 PMXuanwo
05/01/2024, 3:41 AMXuanwo
05/02/2024, 7:01 AMXuanwo
05/02/2024, 7:02 AMXuanwo
05/02/2024, 7:03 AMmarsupialtail
05/02/2024, 10:23 PMming
05/09/2024, 6:38 PMallow_anonymous
argument is getting ignored when I create an S3 service and convert it into an object store. In my builder I've done builder.allow_anonymous().disable_config_load()
to make sure it's not signing credentials or pulling credentials from anywhere that I don't explicitly define in the builder. But when I query this object store (using datafusion, same setup as above) I get the error ERROR: Object Store error: Generic Unexpected error: Unexpected (temporary) at stat, context: { called: reqsign::LoadCredential, service: s3, path: path/to/file.parquet } => loading credential to sign http request, source: error sending request for url (http://169.254.169.254/latest/api/token)
which suggests that it's still loading/signing credentials. I've isolated it to allow_anonymous
because when I do pass in an access key/secret the query works, and I've confirmed that my file is in a public bucket.ming
05/09/2024, 6:39 PMchxlD
05/13/2024, 3:26 PMrust
use bytes::Bytes;
use object_store::path::Path;
use object_store::ObjectStore;
use object_store_opendal::OpendalStore;
use opendal::{services::S3, Builder, Operator};
use std::{fmt::Debug, sync::Arc};
fn register_object_store() {
let builder = S3::from_map(
vec![
("access_key".to_string(), "my_access_key".to_string()),
("secret_key".to_string(), "my_secret_key".to_string()),
("endpoint".to_string(), "my_endpoint".to_string()),
("region".to_string(), "my_region".to_string()),
]
.into_iter()
.collect(),
);
// Create a new operator
let operator = Operator::new(builder).unwrap().finish();
let obj_store = Arc::new(OpendalStore::new(operator));
let path: Path = "data/nested/test.txt".into();
let bytes = Bytes::from_static(b"hello, world! I am nested.");
let res = obj_store.put(&path, bytes.into());
}
When using this put method, inform me its not exist?
txt
no method named `put` found for struct `Arc<OpendalStore>` in the current scope
items from traits can only be used if the trait is in scope;
the following trait is implemented but not in scope; perhaps add a `use` for it:: `use object_store::ObjectStore;
chuckd117
05/14/2024, 12:23 AMNKID00
05/14/2024, 3:09 PM