When you need to open 3 (or more) files in Python3

The problem is I needed to open 3 files:

1. CSV source file to read

2. CSV file to write results on after processing the first file

3. Another CSV file that has metadata resulting from processing the 1st CSV that's different from the 2nd

Solution

There are 2 practical ways to do it. There's a 3rd but it has a very specific use case.

As for Python3, you can basically do this using context managers via the 'with' keyword:

with open('a', 'r') as a_file, open('b', 'w') as b_file, open('c', 'wb') as c_file:    
    do_something()

I believe it this also exist in Python 2.7 but you might need to use the contextlib.nested() function

The other approach is when you need to process files sequentially rather than opening all of them at the same time. Useful if you have variable number of files:

for fname in filenames:
    with open(fname) as f:
        # Process f

You can also use this if you would rather keep the results in memory and only write it to the file when done. 

Reference: 

• https://stackoverflow.com/questions/4617034/how-can-i-open-multiple-files-using-with-open-in-python

Python zipfiles with the cloud is technologic

Working with zip files in Python is straight forward.

Write it

s3connection = self.get_s3_connection()
bucket = s3connection.get_bucket(aws_bucket_name)
k = Key(bucket)
k.key = key.name

k.get_contents_to_filename(file_path)

Cut it, Paste it, Save it

import shutil
shutil.copyfile(self.src_zipfile, self.temp_zipfile)

Load it

zf = zipfile.ZipFile(zip_file)   # turn file into a zipfile object
zf.extractall(path=unpack_path)  # extract all files to path
files = glob.glob(os.path.join(unpack_path, 'list_*.txt'))  #glob glob
for files in files:
    # to stuff to the files here

Check it

# In a test case
zf = zipfile.ZipFile(zip_file)
zf.extractall(path=unpack_path)
files = glob.glob(os.path.join(unpack_path, 'list_*.txt'))
self.assert(len(files),expected_file_count)

Quick - rewrite it

with open(some_file, 'r+') as file:
    #do file rewrite here
    file.seek(0)
    file.write('Quick!')
    file.truncate()

Now here's daftfunk - technologic