Delete orphan files

[sungwy]

Introduce a new API to delete orphan files for a given table

Feature reference: https://iceberg.apache.org/docs/1.5.1/maintenance/#delete-orphan-files

[omkenge]

Hi @sungwy
I would like to work on this ..
Can I ?

[sungwy]

Hey sure thing! I'll assign it to you @omkenge

[omkenge]

Orphan File Deletion in Iceberg Tables
Here's a step-by-step breakdown of the logic behind the process:

1. List All Files in Storage
2. Extract Referenced Files from Table Metadata
3. Identify Orphan Files
   By comparing the list of all files in storage with the list of files referenced by the Iceberg table, the script identifies orphan files.
   These are files that exist in storage but are not part of the current table metadata.
   The comparison is performed by subtracting the set of referenced files from the set of all files in storage.
4. Delete Orphan Files

What is your opinion on this ?
@kevinjqliu @Fokko @sungwy

[kevinjqliu]

That looks generally correct to me. There are a few caveats though. This assumes that the entire iceberg table (metadata and data files) is in a single location and that no other files should exist.

I think a good first step is to figure out all the files belonging to an iceberg table. Given a table, return all metadata and data file paths, including historical lineage, branches, and tags.

[ndrluis]

@omkenge I believe you will need to wait for the merge of #1285. In the meantime, I will work on the partition statistics over the next few weeks. Before that, I believe we will be tracking all the files in the metadata (this needs to be double-checked). With that, you will be able to verify what could be removed.

Another point is the filesystem that will be responsible for scanning the directory. FileIO is not how we solve this, so we will need to use something else. Perhaps OpenDAL would be a good candidate. As a reference, you can see that the Java implementation uses the Hadoop filesystem.

[omkenge]

Hello @ndrluis
I think #1285 is now merged can I start working on this issue.

[ndrluis]

Hello @omkenge, you can start development, but please note that we need the partition statistics. I'll start working on this feature this week. The merge for the orphan files removal implementation will be blocked until we have these statistics, but you can begin the development work.

[omkenge]

Hello @ndrluis @kevinjqliu
Could you plz help me on OpenDal how we can use and integrate this. It will very helpfull for me.
and another thing I just extract the data file from snapshot with
metrics=table.inspect.files()
file_paths = metrics.column("file_path").to_pylist()
is this correct way

[ndrluis]

Hi @omkenge, I don’t have direct experience with OpenDAL, but my suggestion is based on how iceberg-rust is currently using it.

For the implementation, I’d recommend aligning with the Java implementation as a reference. Check out these two key files:

DeleteOrphanFilesSparkAction.java
RemoveOrphanFilesProcedure.java