Wow, look what Google did: <https://github.com/goo...
# maintainers
r
Wow, look what Google did: https://github.com/google/magika
πŸ‘ 1
The model is about 3Mb, and looks embedded in the libs and CLI
m
ohhh cool!
r
Magika is entirely local β€” no external services, no network calls.
Here's what's happening under the hood:
β€’ Google's Magika file type detection library is powered by
ort
, which is a Rust wrapper around Microsoft's ONNX Runtime. This means inference runs on a locally bundled ONNX model, entirely in-process. ort Docs
β€’ Magika uses the high-performance ONNX Runtime for model inference and Tokio for asynchronous parallel processing β€” both of which operate purely on local CPU resources. Google Open Source
β€’ The API works by creating a
Session
that can identify files from their path or from content read directly from memory β€” there's no remote endpoint involved anywhere in that flow. Lib.rs
The model weights are bundled into the crate itself (that's what the
sync.sh
/
gen
tooling in the repo is for β€” baking a new model into the library when one is released). When you call
identify_file
or
identify_content
, it feeds bytes into the local ONNX model and returns results without touching the network.
So you're safe to use it in air-gapped or privacy-sensitive environments. The only "AI" involved is a small neural network running locally via ONNX Runtime.
I think we can use this for the file content detection and not have a GPL license issue on non-Linux OSs
How it actually reads files: Magika's deep learning system processes three sequences of 512 bytes β€” from the start, middle, and end of each file. So it reads ~1.5KB total, regardless of file size. This is what allows it to distinguish things like JSON vs JSONL, C vs C++, or JavaScript vs TypeScript, where magic bytes alone are useless.
cool doge flip 1
m
That’s a good point, I forgot about that issue as it causes some OS behavioural differences if I’m not mistaken?