[magika-lib] Handle non-regular / special files (/dev/null, FIFOs, devices, sockets) as ContentType::Unknown
Context
While running the expanded Python public API test suite (PR #1458, specifically test_special_device_file), we found that magika-lib (rust/lib) does not check metadata.is_file() before opening a path in Session::identify_file (rust/lib/src/session.rs:41-50) and FeaturesOrRuled::extract_file (rust/lib/src/future.rs:59-67).
Current State in rust/lib
let metadata = std::fs::symlink_metadata(&file)?;
if metadata.is_dir() {
return Ok(FileType::Directory);
}
if metadata.is_symlink() {
return Ok(FileType::Symlink);
}
let file = std::fs::File::open(file)?;rust/lib only checks metadata.is_dir() and metadata.is_symlink(), and assumes any other path is a regular file (metadata.is_file()). This causes two issues on Unix special files:
- Character/block devices like
/dev/null:std::fs::File::open("/dev/null")succeeds andmetadata.len()is0, sorust/libclassifies/dev/nullasContentType::Empty(empty).- In Python
Magika(test_special_device_file), non-regular files (!stat.S_ISREG(st_mode)) are never opened; they immediately returnstatus = Status.OK,output.label = ContentTypeLabel.UNKNOWN,dl.label = ContentTypeLabel.UNDEFINED,score = 1.0(i.e.FileType::Ruled(ContentType::Unknown)).
- Named pipes (FIFOs) and sockets:
- Calling
std::fs::File::openon a FIFO with no writer blocks indefinitely, hanging the process.
- Calling
- Divergence between
rust/cli,rust/lib, andpython:- In PR #1464 (
9096fd3),rust/cliaddedensure!(metadata.is_file(), "Not a regular file")inrust/cli/src/main.rs:355, which treats/dev/nulland other special files as an error rather thanContentType::Unknown. - Meanwhile, Python
Magikatreats non-regular files as a validFileType::Ruled(ContentType::Unknown)result (status = Status.OK,output = UNKNOWN,dl = UNDEFINED).
- In PR #1464 (
Proposed Rust API Changes
In rust/lib (Session::identify_file and FeaturesOrRuled::extract_file), after checking metadata.is_dir() and metadata.is_symlink(), check:
if !metadata.is_file() {
return Ok(FileType::Ruled(ContentType::Unknown)); // or FeaturesOrRuled::Ruled(ContentType::Unknown)
}without calling std::fs::File::open. This prevents magika-lib from misclassifying /dev/null as empty or hanging on FIFOs, and allows rust/pyo3 (and rust/cli, if desired) to rely directly on magika-lib without custom special-file checks in the bindings layer.
Source: google/magika