When you need to open 3 (or more) files in Python3

The problem is I needed to open 3 files:

1. CSV source file to read

2. CSV file to write results on after processing the first file

3. Another CSV file that has metadata resulting from processing the 1st CSV that's different from the 2nd

Solution

There are 2 practical ways to do it. There's a 3rd but it has a very specific use case.

As for Python3, you can basically do this using context managers via the 'with' keyword:

with open('a', 'r') as a_file, open('b', 'w') as b_file, open('c', 'wb') as c_file:    
    do_something()

I believe it this also exist in Python 2.7 but you might need to use the contextlib.nested() function

The other approach is when you need to process files sequentially rather than opening all of them at the same time. Useful if you have variable number of files:

for fname in filenames:
    with open(fname) as f:
        # Process f

You can also use this if you would rather keep the results in memory and only write it to the file when done. 

Reference: 

• https://stackoverflow.com/questions/4617034/how-can-i-open-multiple-files-using-with-open-in-python

Notes on Solving Simple Vehicle Routing with Python and Google Maps only

Simple Vehicle Routing is a problem related to finding the "optimum" path between a set of locations. The rub here is dealing with real-world addresses and road data (ie. traffic, condition, etc.). Google documentation (or propaganda) will say this is easy but that's slightly misleading.

What I've learned:

 1. Setting up to use the Google Maps APIs (yes APIs; we are going to use most of the it) needs a billing account or credit card. Although, it will not cost you anything because Google is giving you usd$200 month credit. So unless, you exceed 200 bucks worth of usage, it won't cost you anything BUT it's a good idea to set a limit just in case.

2. Jupyter Notebooks is handy to figure out and test stuff. It's just a little finicky getting it to run. For example:

• I had to rollback ipywidgets to 7.7.2 from the latest 8.0.9 to get gmaps to work
• I had to change python versions from 3.11 to 3.9 because scipy was being a bitch when installing ortools. Ortools has dependencys to scipy. Luckily I already use pyenv so this was relatively painless.

Jupyter Notebook running with gmaps, OR-Tools

Next step is figuring out how get what I've learned into a REST API for delivery/pickup app. I thinking, use Starlite + Rethinkdb.

Ref: 

• https://woolpert.com/managing-simple-vrp-with-google-maps-platform/
  
• https://developers.google.com/optimization/routing/vrp
• https://jupyter-gmaps.readthedocs.io/en/latest/install.html
• https://stackoverflow.com/questions/72371859/attributeerror-module-collections-has-no-attribute-iterable

Flutter + Python = Flet

Flet is this pretty cool framework that allows someone to build an app on web, mobile and desktop quickly. It uses the Flutter framework for the UI stuff to get it to look good and work on any platform. 

It says that its language agnostic but it currently only supports Python. Does that make it a Python framework?

Anyway, I've been playing around with it and I like it. 

Things that I like:

1. Good looking apps created quickly

2. Everything is in Python even the UI; quite similar how Flutter does it, in fact. 

3. Next-to-nothing code change to target different platforms; so, to get a desktop app to run on a web browser is a just a named function parameter

flet.app(target=main) # run it as a desktop
flet.app(target=main, view=flet.WEB_BROWSER) # run it on a web browser/web app

Things that's missing:

1. No hot reload (but it's on the roadmap so that's something to look forward to)

2. Not all of the Flutter widgets are available like the Hero widget

Overall, I'm having a grand old time trying out Flet. 

Enforce row count limits on ManyToManyField rows in Django models

I had a requirement where ModelA needs to limit how many ModelBs it can have. They are linked via a ManyToManyField using a through model.

It looks like this:

 class ModelA(models.Model):  
   name = models.Charfield(max_length=255)  
   b_limit = models.PositiveSmallIntegerField(help_text="Max number of Bs we can have")  
   model_b_set = models.ManyToManyFields('ModelB', through="ABJoinTable")  

 class ModelB(models.Model):  
   some_field = models.Charfield(max_length=255)    
   model_a_set = models.ManyToManyFields('ModelA', through="ABJoinTable")  

 class ABJoinTable(models.Model):   
   model_a = models.ForeignKey("ModelA", on_delete=models.CASCADE)  
   model_b = models.ForeignKey("ModelB", on_delete=models.CASCADE)  
   is_suspended = models.BooleanField(default=False)  

The through model just contains extra information is NOT the key to enforcing the rule. Remember, we want to limit how many ModelB rows ModelA can have. How many ModelB's ModelA can have it controlled by the "b_limit" field. 

The solution is to use the m2m_changed signal. 

from django.db.models.signals import m2m_changed  
from django.core.exceptions import ValidationError

def enforce_modelA_B_limit(sender, **kwargs):  
   modelA = kwargs['instance']  
   if modelA.model_b_set.count() >= modelA.b_limit:  
    raise ValidationError("ModelA has too many ModelBs")  

m2m_changed.connect(enforce_modelA_B_limit, sender=ModelA.model_b_set.through)  

You can place this wherever a signal function is valid. I just placed mine in the same place where the models are declared.

Why a signal?

If your gut feeling was to override the `save()` or `clean()` methods then those won't work. You cannot check the count for the ManyToManyField until you have save the primary model - ModelA. You'll get an exception saying something like: `ModelA needs to have a value for field "id" before this many-to-many relationship can be used.`.

Testing Tip

In writing a test case for this, you can use the `self.assertRaises()` context to catch the error. It's bad practice if you do a `try-catch` block to see if the rule works. The test case would look like this:

 def test_modelA_B_limit_rule(self):  
   # we assume that a setUpTestData() method exist  
   # we also assume that ModelA is limited to 1 ModelB so adding a second ModelB should throw an
   # exception
   sample_b = ModelB.objects.create(some_field="test")  
   modelA_instance = ModelA.objects.get(pk=1)  
   with self.assertRaises(ValidationError):  
     modelA_instance.model_b_set.add(sample_b) # should throw  

There's nothing to change if you have Django Rest Framework project. Your API will throw a 400 error if they violate the limit.

Refs: https://stackoverflow.com/questions/20203806/limit-maximum-choices-of-manytomanyfield

Fixing a Django drf-yasg "serializer_class required" AssertionError

Sometimes you have an API view (ie. GenericAPIView) class without a serializer. This causes the drf-yasg generator to throw an AssertionError exception saying class should either include a 'serializer_class' attribute, or override the 'get_serializer_class()' method.

You can fix this by adding the following to the view class:

 def get_serializer(self, *args, **kwargs):  
     pass  
 def get_serializer_class(self):  
     pass  

This should pass the drf-yasg serializer check.

Reference

• https://github.com/axnsan12/drf-yasg/issues/754

Django tests on an Azure Pipeline

Source: https://gist.github.com/killertilapia/c9e7635807e596f20a16123ceed9c48d

Running Django tests on Azure is a mix bag of confusing documentation and outdated examples. When We did this I can't shake the feeling that Python was a second class citizen in Azure. 

Stuff that bugged me OR caught me flat-footed

1. Environment Variables. I get that having many options supported for declaring variables is cool but I wish Azure followed one of Python's tenets and that's "There should be one— and preferably only one —obvious way to do it.!". 

Another thing that bugged me was how Azure allows declaring variables for the pipeline: why allow a `.` (period) in the declaration then explain that gets replacement with `_` (underscore) and the rest of the text capitalized under certain conditions. 

Why couldn't we just stayed with YML? I never did like idea of "scripts within scripts". Again, Python's tenets: "There should be one— and preferably only one —obvious way to do it.!". 

2. vmImage version. This isn't really an Azure issue but more of a Python test issue. In order to run the tests I wanted: 

python manage.py test --testrunner xmlrunner.extra.djangotestrunner.XMLTestRunner --no-input --settings=settings.testing

I had to create an environment for the test to run on. So step before this was to install my environment prerequisites which is directly influenced by the vmImage version. At first, I was using: ubuntu-16.08 which was causing a Django migration error due to unsupported feature. We quickly figured out that the sqlite version that came with 16.08 was older. We also had to bump up the python version to 3.8. 

If you're basic and prefer to use a UI then Azure is probably your thing. But an old school guy like me who prefers automated scripts, Azure's documentation and philosophy to cloud is an inconvenience. 

Downtime Investigations: Pipenv vs Poetry

Python projects start around a virtual environment. This helps organizing project dependencies and builds are deterministic. In this area there are two maturing tools Pipenv and Poetry. I've been a long time user of pipenv but it's never a bad idea to see what the other side has to offer. 

Handling Packages

Both Pipenv and Poetry organize project dependencies with a separate file for production and development. ie: requirements.txt, requirements-dev.txt

Pipenv uses a Toml file called Pipfile while Poetry uses a similar Toml file called pyproject.toml. Interesting side note, unlike Pipfile, pyproject.toml follows PEP 518. 

The difference here, I discovered, is that Pipfile is smaller in scope vs pyproject.toml. The pyproject.toml can be contain configs for supports tools like flake8 or publishing info if you're going to put the project up to pypi. You need a separate file, like say setup.py, in Pipenv to publish your project/package. 

Adding dependencies

Pipevn's Install command installs all dependencies if you don't specify a package. Poetry decided to have separate commands for add a new dependency and installing existing ones. In effect, Poetry ask the dev to be more explicit on what he's try to do. 

Poetry also has more info in the terminal vs Pipenv which is fairly spartan when installing something.

Pipenv installing pytest

Poetry installing pytest

Uninstalling dependencies

A thing that I discovered, Poetry uninstalls sub-dependencies. Pipenv does not. With pipenv you have to do something like pipenv uninstall [some package] && pipenv clean to do what Poetry does from a single poetry remove [some package]

Wrap up

Both Pipenv and Poetry goal's are to make dependency management easier and building projects more consistent. While pipenv has more broader support but I think that's just because it's older than poetry. Poetry has some interesting features like having configs in it's project Toml and uninstalls sub-dependencies. 

At some point in the feature, I think I'd like poetry in my professional projects. 

Django-celery Error in Calling apply_async() - takes 1 positional arguement but xx were given

This error confused me initially and the Celery documentation wasn't directly helpful. 

 Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "/Users/jaypax/.local/share/virtualenvs/server-9an_1rEM/lib/python3.6/site-packages/celery/app/task.py", line 518, in apply_async  
   check_arguments(*(args or ()), **(kwargs or {}))  
 TypeError: run_scraper_one() takes 1 positional argument but 40 were given  

I called my task as run_scraper_one.apply_sync(args=('keywords here'), countdown=5). 

The run_scraper_one() method is decorated with @shared_app. So this should work. Should. But apparently after digging around: here and here, I figured out that it wants a list or tuple. 

So, the correct way to invoke the task is: 

run_scraper_one.apply_async(("keyword here",), countdown=5)

Fixed.

Scrapy a JS heavy website using Selenium

Scrapy doesn't really like JavaScript heavy websites especially the ones that load the rest of the HTML via a secondary requests using JavaScript.

To overcome this you either use Splash or Selenium. Unfortunately, Splash is no longer supported. It still works but moving forward, it's going to be Selenium. 

The good news here is that Scrapy already supports Selenium via a middleware: scrapy-selenium.

Steps to use scrapy-selenium:

1. Download a Selenium Driver. For example for Firefox get gecko. I'm assuming you have the browser also installed. 

2. Add the following settings to the settings.py file:

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'path/to/gecko'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'path/to/firefox binary'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

For example (on Windows):

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'c:\\Tools\\geckodriver.exe'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'c:\\Program Files\\Mozilla Firefox\\firefox.exe'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

3. In the spiders, just replace the Request() calls to SeleniumRequest()

4. Add a wait_until test on the SeleniumRequest() to would look like:

SeleniumRequest(url=url, 
                callback=self.parse, 
                wait_time=5,
                wait_until=EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.search-results div.search-cell')))

In this one, we wait for the max of 5 secs or until the element, selected by class to be found in the HTML source. After that, we can use scrapy selectors to find the things we want.

So that's it.

Testing for many values in a Python list

There will be times that you'd want to test for membership of multiple values in a list.

Of course, we could solve this my writing the equivalent number of loops for every value we are testing. That's a nope.

Fortunately Python provides a much cleaner solution:

>>> all(x in ['b', 'a', 'foo', 'bar'] for x in ['a', 'b'])

Also there are other options:

>>> set(['a', 'b']).issubset(set(['a', 'b', 'foo', 'bar']))
True
>>> {'a', 'b'} <= {'a', 'b', 'foo', 'bar'}
True

Reference:

https://stackoverflow.com/questions/6159313/can-python-test-the-membership-of-multiple-values-in-a-list

Handling non-well-formed HTML in Scrapy with BeautifulSoup

With Scrapy, we can deal with non-well-formed HTML is many ways. This is just one of them.

BeautifulSoup has a pretty nifty feature where it tries to fix bad HTML like replacing missing tags. So if we put BeautifulSoup in the middle then whatever we get from a site is fixed before we parse it with Scrapy.

Fortunately, all we have to do is pip install Alecxe's scrapy-beautifulsoup middleware.

pip install scrapy-beautifulsoup

Then we configure Scrapy to use it from settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_beautifulsoup.middleware.BeautifulSoupMiddleware': 400
}

BeautifulSoup comes with a default parser named 'html.parse'. We can change it.

BEAUTIFULSOUP_PARSER = "html5lib"  # or BEAUTIFULSOUP_PARSER = "lxml"

HTML5 is the better parser IMO but it has to be installed separately.

pip install html5lib

Using ElementTree(xml) with Scrapy to deal with hard to deal HTML

While Scrapy's Selectors like xpath and css are powerful there are some cases that make them cost to much effort.

An example with irregular HTML text like this:

 ['<a href="https://www.blogger.com/u/1/misc/cst2020d.html"><b>Dimensions</b></a>',  
  '<a href="https://www.blogger.com/u/1/misc/cst2020s.html"><b>Typical Circuit</b></a>',  
  '<a href="https://www.blogger.com/u/1/misc/cst2020t.html"><b>Temperature vs Current</b></a>',  
  '<a href="http://www.blogger.com/Search.aspx?arg=somepart%20ic2020" target="_blank"><b>3D\xa0model</b></a>']  

This is a sample result of calling the extract() method with Scrapy's selector. The parts we want here is the href and the link text. We want to get those.

We can:

1. Do multiple Scrapy selector calls to get the data we need or
2. Do a single Scrapy selector call and process it via XML

I went with #2. Dealing with HTML as XML should be relatively easy. Besides Python already has a way for working with XML via the ElementTree XML API.

So the Python code to solve the problem is short and simple:

import xml.etree.ElementTree as ET
....
class MySpider(scrapy.Spider):
....
    for link_item in our_raw_links:
        root = ET.fromstring(link_item)
        href = root.attrib['href']
        anchor_text = root[0].text

        cap_links.append({'text': anchor_text, 'href': href})

And Voila!

Troubleshooting Cmder Terminal in Visual Studio Code

If you didn't know, you can customize the terminal for VScode. In my case, I wanted to use cmder because Windows terminals suck; both of them cmd and powershell.

Fortunately, cmder has a guide for integrating it to VScode.

The interesting thing is when I tried using it with my Python workflow which includes virtual environements via Pipenv and I ran into a couple of problems.

Problem #1 Missing posh-git

Symptom Spits out a warning telling you that you are missing posh-git - 'Install-Module posh-git' and restart cmder"

Fix #1 Run from the admin level PShell:  Install-Module posh-git

You might also get a error saying it might have an overlap with some other extension like say 'TabExpansion'.

Fix #2 Either run Fix#1 with the -AllowClobber or -Force option OR run it with a -Scope option: Install-Module posh-git -Scope CurrentUser

----

Problem #2 FunctionNotWritable (Cannot write to function prompt..)

Symptom Spits out the error; Also notice that your terminal prompt isn't prefixed by the activated virtual environment. Only happens when your doing Python with virtual environments.

Fix #1 Remove or comment out the -Options ReadOnly option on the cmder profile.ps1. Details here.

Random quirk with Windows, Git and an executable Bash script

The context of the problem was that I was making a Docker container with Linux image. It's to be deployed to a cloud service and I'm working from Windows machine. Of course the container would need code to run. To facilitate this, I wrote a basic bash script - scrape.sh.

On Windows, the scrape.sh file already is executable

For disclosure, I'm working with Scrapy here.

From my Windows terminal, the scrape.sh script seems to be already executable. I thought I was fine and I did the whole git cycle of add, commit and push.

The problem shows up when the container tries to run. It will spit out an error saying that it can't execute the scrape.sh script because it's lacking rights. It seems that git commits my bash script as an ordinary file not as an executable despite what I saw on the Windows terminal. The fix though was quite easy.

$ git add --chmod=+x -- afile
$ git commit -m"afile is now executable"

Relevant link:

• Stackoverflow
• Git-docs

Installing Scrapy on Windows

Data is the new oil they say and you want to start scraping sites for data. Fine! And since you are a Python developer you'd want to use Scrapy. Unfortunately for you you are a Windows user and errors abound.

Anyhow, you run pip install scrapy and you run into an error:

Scrapy failing to pip install in Windows

The error here is one of dependencies of Scrapy which is Twisted. Fortunately this is fixable.

The first thing we're going to do is download the "binary" wheel file for Twisted. There's a trick here though, YOU MUST DOWNLOAD THE CORRECT ONE THAT MATCHES YOUR PYTHON VERSION. So if you are on Python 3.6, you're looking for something that reads like Twisted-18.9.0-cp36-cp36m-win32.whl; if Python 3.7 then you're looking for something with cp37-cp37m. You get the idea.

The 32-bit or 64-bit probably also matters but I didn't test it.

Once you have that file downloaded somewhere, you can then do a pip install .

Try installing Scrapy again and you should be good to go.

As an added bonus, this seems to fix other Python packages in Windows that require the Visual Studio Windows C++ SDK like the mysql-python.

Tip for Pipenv users:

1. pipenv shell
2. pip install
3. pipenv sync

Pyenv folder (.venv) with Pipenv

Pipenv, by default will use your virtualenv folder location if you set that environment variable - which probably be a .virtualenv or .env folder in your home directory.

But sometimes you would like to have the virtualenv folder within the project folder - ie. project/.venv No sweat, pipenv supports this workflow if set the PIPENV_VENV_IN_PROJECT environment variable.

For Macs: $ export PIPENV_VENV_IN_PROJECT=1

For Windows: > set PIPENV_VENV_IN_PROJECT=1

Do note that in Windows, you can also use setx.

Then we can run either a: pipenv install or pipenv --three to get started.

Understanding the two kinds of smart contracts

Stellar is a platform built with Blockchain that enabled someone to build a financial system like say a payment system. One feature on Stellar is Stellar Smart Contracts (SSC) and I've been trying to wrap my head around it.

So what is a smart contract? An SC is just an agreement between 2 or more parties formalised over a computer network. Think paper contracts but with a computer network as a witness. The catch is that there are TWO kinds of smart contracts - Turing complete OR non-turing.

Turing complete SCs like Ethereum's Solidity include code that EXECUTES ON THE CONTRACT when given certain inputs. What can be executed is still subject to constraints on the contract like say time - ie. You can execute this contract after 48 hours.

Non-Turing complete SCs like Stellar's do not include code with the contract. So you have to execute the contract on your side - ie. your own server. But similar to the other type of SC, what can be executed is still subject to constraints.

Turing complete SCs are like self contain black boxes with buttons and sensors. It does stuff on it's own based on what button is pressed or what the sensors pick up.

Non-Turing SCs on the other hand are like those hand-crank jack-in-box toys, who (when, how, etc.) can crank it defined is identified on the toy. On it's own it doesn't do much. It just sits there.

So what use cases are talking about?

Non-Turing SCs seem to be pretty good for escrow type transactions. Stellar's SC examples are in fact, escrows.

Turing complete SCs seem to fit into long term, repeating type transactions like retirement payments.

Which one to use? I can't say. I've been learning Stellar via a Python SDK. It's been interesting so far.

Why use the middleware approach for Django analytics?

Because of either proxy servers, ad blockers (like ABP, uBlock, etc.), or browser settings that stop JS scripts. Depending on the level of users and how much value they place on their privacy, they can outright BLOCK all forms of analytics. This ends up with our analytics data being unreliable or at least viewed with less confidence.

But why the middleware?

Django middleware is a good place because it's called for all request and response cycle.

The upside to this is:

1. It's easy to customise. Django middleware is just a plain Python class with some methods. 
2. The Django middleware is well-documented. 
3. Allows us to setup whatever rules like exclude errors and only track HTTP 200 responses.
4. We can setup tracking to be asynchronous. This way, our pages are not at the mercy of an external resource.

Writer's note: If your analytics needs are simple, you can install django-google-analytics and follow Usage #2 - Middleware + Celery.

There are a couple of downsides to this though:

1. We could be impacting performance in a big way if the analytics middleware is doing too many things or worst, is misconfigured. 
2. Not as bad as #1 but if we went down the asynchronous path, we will end up with a Celery server with some message broker like RabbitMQ on the tech stack. Another thing we'll have to manage. 

TL:DR For analytics with Django, HTML tag slows down pages and could be blocked; better to use a middleware approach.

Using Django forms to validate POST data in Rest Framework

Django Rest Framework already has a couple of ways to validate data. I believe the "preferred" way is to validate data from serializers. For example:

from rest_framework import serializers

class SomeSerializer(serializers.ModelSerializer):

    def validate(self, data):
        errors = {}
        year_built = data.get('year_built')
        year_rennovated = data.get('year_rennovated')
        
        if year_rennovated < year_built:
            errors['error'] = "You can't renovate something that doesn't exist!"
            raise serializers.ValidationError(errors)
            
        return data

But you can use Django forms, especially if you have existing ones that have the same "shape" of your post data.

Here's an example form:

from django import forms

class BuildingForm(forms.Form):

    year_built = forms.IntegerField(required=True)
    year_rennovated = forms.IntegerField(required=True)
                            
    def clean(self):
        cleaned = super(BuildingForm, self).clean()
        year_built =  cleaned.get('year_built')
        year_rennovated =  cleaned.get('year_rennovated')

        if year_rennovated < year_built:
            raise forms.ValidationError(u'Oops!')

And then in our API View:

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

class BuildingView(APIView):
    def get(self, request):
        model_form = forms.BuildingForm(request.data)
        if model_form.is_valid():
            data = serializers.someSerializer(obj).data
            return Response(data, status=status.HTTP_200_OK)
        else:
            return Response({'errors': model_form.errors}, status=status.HTTP_400_BAD_REQUEST)

Django lazily evals URL patterns so I thought the "include" function was broken

Little known things sometimes kick your ass and you'll need to call a friend.

Until today, I didn't realise that Django lazily evaluates it's URLs. I discovered this when the code me and my team were working on had a cascading URL include. So we had something like:

# main urls.py
urlpattern = [ 
   url(r'^path/', include('app1.urls'),
]

# 2nd url - app1's url.py
urlpattern = [
   url(r'^app1/', include('app1.subapp.urls'),
]

# 3rd url - subapp for app1 urls.py
urlpattern = [
   url(r'^subapp/$, actual_view.as_view()),
]

Problem started when I assumed that the full URL path will show in the debug page on the browser. I was looking for /path/app1/subapp/. I only saw the /path listing.

This is where I thought that Django was not registering the rest. It never dawned on me that by just doing "localhost:8000/path/" will allow me to see the rest of the URL pathing. Django just shows the same or next urls not the urls 2 levels in.

Thank you Eric (@riclags) for pointing out that fact. 

I'm a dumbass sometimes.