Large Dataset Filtering with Python

A small filter is easy to build when the file fits in Excel.

The problem changes when the input is measured in gigabytes. Loading a 20 GB or 200 GB CSV into memory is not a reasonable default, and asking a non-technical user to write a command-line query may not solve the operational problem.

I built a public Python project to explore that gap.

The problem

Imagine a file containing city records. The user needs rows where the city contains “New York” or equals “Chicago”, and the result has to be exported for the next step.

The filtering rule is simple. The difficult parts are:

• the source may not fit in memory;
• the user still needs to define and review the condition;
• identifiers and values must survive the round trip;
• the output has to be understandable outside Python.

Processing the file in chunks

The original project uses pandas chunking.

Instead of loading the full CSV, Python reads a manageable block, applies the condition, keeps matching rows and continues with the next block. Peak memory usage is then related to the chunk size rather than the whole source file.

The basic shape is straightforward:

for chunk in pandas.read_csv(path, chunksize=chunk_size):
    matches = chunk.query(filter_expression)
    write_or_collect(matches)

Real code needs more care than this example. Column types, delimiters, encodings, malformed rows and output strategy all need explicit decisions.

A small interface for a non-developer

I used Tkinter for the desktop interface.

The goal was not to build a polished commercial application. It was to let a user choose a file, define filtering conditions and export CSV or XLSX results without editing Python.

That choice also created a useful design constraint: if a filtering rule cannot be explained clearly in the interface, it probably is not clear enough in the implementation either.

What I would change today

The project still represents the core idea, but I would separate the parts more explicitly now:

1. a parsing and validation layer;
2. a filtering engine with named rules;
3. an output writer with row counts and checks;
4. a thin interface on top.

Depending on the file and deployment, I would also evaluate DuckDB, Polars or an actual database instead of assuming pandas is always the right engine.

The important point is not a library preference. It is making memory limits, data types, filter logic and output validation visible.

What was verified

This is an independent public project, not a client case with a measured commercial result.

The code demonstrates chunked reading, user-defined filtering and CSV/XLSX output. It shows a reproducible approach to a class of problems that otherwise turns into repeated manual extracts.

It does not claim that every 200 GB file should be processed by this desktop tool. At that scale, profiling the real file and choosing the correct engine is part of the job.

Source

The project is available on GitHub. I also published the original short article on Medium.

The larger lesson still holds: “filter this file” is not only a query. It is a small data workflow, and it needs explicit rules, resource limits and a way to verify the output.