tag:blogger.com,1999:blog-82774742349314011192024-02-19T19:44:37.577+08:00angwebampgonnakillsomebodyBlog. Repository of ideas/solutions. Web Developer. Data Labandero. Python. Java. TypeScript. C#. Flutter apps. VB6 free since 2003!
Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.comBlogger221125tag:blogger.com,1999:blog-8277474234931401119.post-57767801686193391452023-07-10T07:09:00.001+08:002023-07-10T07:14:14.470+08:00When you need to open 3 (or more) files in Python3The problem is I needed to open 3 files:1. CSV source file to read2. CSV file to write results on after processing the first file3. Another CSV file that has metadata resulting from processing the 1st CSV that's different from the 2ndSolutionThere are 2 practical ways to do it. There's a 3rd but it has a very specific use case.As for Python3, you can basically do this using context managers Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-11383720768765244332023-03-30T10:58:00.002+08:002023-03-30T10:58:15.121+08:00Notes on Solving Simple Vehicle Routing with Python and Google Maps onlySimple Vehicle Routing is a problem related to finding the "optimum" path between a set of locations. The rub here is dealing with real-world addresses and road data (ie. traffic, condition, etc.). Google documentation (or propaganda) will say this is easy but that's slightly misleading.What I've learned: 1. Setting up to use the Google Maps APIs (yes APIs; we are going to use most of the itJaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-13386212268582922802023-02-13T14:40:00.002+08:002023-02-13T14:40:26.338+08:00Notes on Different Docker image tagsFull ImagesThese are images based on the most latest release of an operating system. For example, if you talking about the `python:latest` then this would be the most recent stable Debian OS release. These images are usually the one's we use at the start of a project.The catch with these images is that they are usually pretty big. The latest python image clocks at around 4GB. If image size is notJaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-55037433208988305502023-01-11T11:28:00.004+08:002023-01-11T11:31:19.680+08:00Notes for Setting up Django for ASGI with Gunicorn, Uvicorn, Docker and WebsocketsGunicorn is a robust web server that has monitoring and automatic restarts. Perfect for production deployments. Uvicorn is an ASGI server based on uvloop and httptools, with emphasis on speed. ASGI means Asynchronous Server Gateway Interface intended for async-capable Python web servers, frameworks and apps. The other side of the coin is WSGI. TL:DR: ASGI = asynchronous, WSGI=Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-88441448611269101122022-12-21T13:49:00.008+08:002022-12-30T07:02:08.955+08:00Dumping and Restoring a DB from host into a Docker container or backAssuming you have the sql (ie. backup.sql) file on hand:To create a backup of a mysql DB (ie. dumping it):$ docker exec CONTAINER_NAME /usr/bin/mysqldump -u root --password=root DATABASE > backup.sqlTo restore a mysql dump (for example):$ cat backup.sql | docker exec -i CONTAINER_NAME /usr/bin/mysql -u root --password=root DATABASE Add the --force or -f flag to ignore errors$ cat Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-33902601074466567802022-11-17T10:30:00.002+08:002022-11-17T10:30:46.218+08:00Fixing bad video files with ffmpegFFmpeg is an open-source collection of tools for processing video and audio files. It allows to convert video and audio files from one format into another, resize videos, stream audio and video, and perform various other actions with media files.FFmpeg is quite easy to interact with Python via bindings but it's also a CLI which what I used to fixed bad video. The simplest command we canJaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-57179692130579620792022-10-06T10:13:00.001+08:002022-10-06T10:13:34.728+08:00Django utils and misleading timezonesDate and time are difficult things to get right in web programming. It has many factors affecting it like the server time zone, the users' time zone, a serverless function calling from a different time zone, you set `USE_TZ` to true in your django settings, etc. I had an interesting problem where my django site had to use `US/Arizona` as the default time zone. At first glance, using `Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-71741065031798576482022-08-17T21:31:00.000+08:002022-08-17T21:31:39.326+08:00Flutter + Python = FletFlet is this pretty cool framework that allows someone to build an app on web, mobile and desktop quickly. It uses the Flutter framework for the UI stuff to get it to look good and work on any platform. It says that its language agnostic but it currently only supports Python. Does that make it a Python framework?Anyway, I've been playing around with it and I like it. Things that I like:Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-1447405978417988002022-06-21T12:38:00.000+08:002022-06-21T12:38:01.317+08:00A guide to better google-fu in 8 formsGoogle is a powerful tool but 98% of the people on the Internet suck at it. Google search can do more than just typing in a word and pressing enter.Here's the 8 tips:"Quotation Marks" Put quotes around search terms to let you search exactly for that word.Example: "Sara Duterte"Gives you all Sarah Duterte search results without just "Sarah" or just "Duterte".- DashesUse dashes to exclude a term Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-27058946686480698642022-06-14T15:37:00.000+08:002022-06-14T15:37:32.728+08:00Enforce row count limits on ManyToManyField rows in Django modelsI had a requirement where ModelA needs to limit how many ModelBs it can have. They are linked via a ManyToManyField using a through model.It looks like this:
class ModelA(models.Model):
name = models.Charfield(max_length=255)
b_limit = models.PositiveSmallIntegerField(help_text="Max number of Bs we can have")
model_b_set = models.ManyToManyFields('ModelB', through="ABJoinTable")Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-55166788506870663692022-06-09T14:54:00.006+08:002022-06-09T14:54:59.826+08:00Fixing a Django drf-yasg "serializer_class required" AssertionError Sometimes you have an API view (ie. GenericAPIView) class without a serializer. This causes the drf-yasg generator to throw an AssertionError exception saying class should either include a 'serializer_class' attribute, or override the 'get_serializer_class()' method.You can fix this by adding the following to the view class: def get_serializer(self, *args, **kwargs):
pass
def Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-21453179726176947102021-12-06T18:03:00.001+08:002022-03-26T19:01:09.624+08:00When Django 4.00rc-1, pipenv, djstripe and docker decide to pull the rug from under youSometimes you just need to rebuild your docker containers and that's where the problem started. When my containers came back, it wouldn't start because of a TypeError being thrown. File "/usr/local/lib/python3.8/site-packages/djstripe/models/__init__.py", line 25, in <module>
from .core import (
File "/usr/local/lib/python3.8/site-packages/djstripe/models/core.py", line 30, Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-35128477264206987552021-10-22T07:25:00.000+08:002021-10-22T07:25:05.143+08:00Updating powershell with powershellTo update powershell with powershell you, run the command: iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"That's it. I'm using Windows Terminal btw.NOTE: This does not update the modules external to PS. You'll have to update those on their own.Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-12005635319060372782021-05-07T07:30:00.008+08:002021-05-07T07:33:16.289+08:00Django tests on an Azure PipelineSource: https://gist.github.com/killertilapia/c9e7635807e596f20a16123ceed9c48dRunning Django tests on Azure is a mix bag of confusing documentation and outdated examples. When We did this I can't shake the feeling that Python was a second class citizen in Azure. Stuff that bugged me OR caught me flat-footed1. Environment Variables. I get that having many options supported for declaring Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-11622512006355014902020-11-25T11:37:00.005+08:002020-11-25T11:39:09.143+08:00Downtime Investigations: Pipenv vs Poetry Python projects start around a virtual environment. This helps organizing project dependencies and builds are deterministic. In this area there are two maturing tools Pipenv and Poetry. I've been a long time user of pipenv but it's never a bad idea to see what the other side has to offer. Handling PackagesBoth Pipenv and Poetry organize project dependencies with a separate file for Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-90366183632865196412020-09-03T15:16:00.004+08:002020-09-03T15:16:41.921+08:00Django-celery Error in Calling apply_async() - takes 1 positional arguement but xx were givenThis error confused me initially and the Celery documentation wasn't directly helpful. Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jaypax/.local/share/virtualenvs/server-9an_1rEM/lib/python3.6/site-packages/celery/app/task.py", line 518, in apply_async
check_arguments(*(args or ()), **(kwargs or {}))
TypeError: Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-21780461769835382912020-03-31T14:04:00.001+08:002020-03-31T14:04:23.571+08:00Scrapy a JS heavy website using SeleniumScrapy doesn't really like JavaScript heavy websites especially the ones that load the rest of the HTML via a secondary requests using JavaScript.
To overcome this you either use Splash or Selenium. Unfortunately, Splash is no longer supported. It still works but moving forward, it's going to be Selenium.
The good news here is that Scrapy already supports Selenium via a middleware: Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-24168885718849574212020-02-10T11:03:00.000+08:002020-02-21T10:03:38.374+08:00Fixing that CosmosDB Error=2: The index path corresponding to the specified order-by item is excluded.This bug needs three things:
You're using Azure CosmosDB (I know, I don't like it too)
Have a Mongoose query with a sort option against a ..
Field that inside a sub document.
The query option in question is `{ sort: req.query.order || '-metaData.inserted_at' }`. The metaData.inserted_at field is just a date field. MetaData is just a plain object that has a couple of date fields tracking Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-67984078334089933872019-11-13T14:35:00.000+08:002019-11-13T14:35:24.333+08:00Testing for many values in a Python listThere will be times that you'd want to test for membership of multiple values in a list.
Of course, we could solve this my writing the equivalent number of loops for every value we are testing. That's a nope.
Fortunately Python provides a much cleaner solution:
>>> all(x in ['b', 'a', 'foo', 'bar'] for x in ['a', 'b'])
Also there are other options:
>>> set(['a', 'b']).Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-44366126804267497552019-10-28T08:42:00.000+08:002019-10-28T08:42:40.479+08:00Keeping a Forked Repo up to dateKeep your fork up to date by tracking the original "upstream" repo that you forked. To do this, you'll need to add a remote:
# Add 'upstream' repo to list of remotes
git remote add upstream https://github.com/UPSTREAM-USER/ORIGINAL-PROJECT.git
# Verify the new remote named 'upstream'
git remote -v
Whenever you want to update your fork with the latest upstream changes, you'll need to first Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-8743326982512451122019-08-29T11:33:00.000+08:002019-08-29T11:33:12.413+08:00Handling non-well-formed HTML in Scrapy with BeautifulSoupWith Scrapy, we can deal with non-well-formed HTML is many ways. This is just one of them.
BeautifulSoup has a pretty nifty feature where it tries to fix bad HTML like replacing missing tags. So if we put BeautifulSoup in the middle then whatever we get from a site is fixed before we parse it with Scrapy.
Fortunately, all we have to do is pip install Alecxe's scrapy-beautifulsoup middleware.
Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-33503985925776144402019-06-26T11:58:00.003+08:002022-12-13T09:18:51.333+08:00Gracefully dealing with different SSH keys for different domains or accountsThis problem often happens when you mix your personal ssh keys along with your company's and you work off a single workstation or laptop.
Even if the user and host are the same, they can still be distinguished in the ~/.ssh/config.
Host gitlab.com
HostName git.company.com
User git
IdentityFile /home/whoever/.ssh/id_rsa.alice
IdentitiesOnly yes
Host kitlab.com
HostName git.company.comJaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-27850080919846501752019-05-21T08:58:00.001+08:002019-05-21T08:58:49.959+08:00Using ElementTree(xml) with Scrapy to deal with hard to deal HTMLWhile Scrapy's Selectors like xpath and css are powerful there are some cases that make them cost to much effort.
An example with irregular HTML text like this:
['<a href="https://www.blogger.com/u/1/misc/cst2020d.html"><b>Dimensions</b></a>',
'<a href="https://www.blogger.com/u/1/misc/cst2020s.html"><b>Typical Circuit</b></a>',
'<a Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-15860007558830629532019-04-10T10:49:00.001+08:002019-04-10T10:49:56.560+08:00Troubleshooting Cmder Terminal in Visual Studio CodeIf you didn't know, you can customize the terminal for VScode. In my case, I wanted to use cmder because Windows terminals suck; both of them cmd and powershell.
Fortunately, cmder has a guide for integrating it to VScode.
The interesting thing is when I tried using it with my Python workflow which includes virtual environements via Pipenv and I ran into a couple of problems.
Problem #1&Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0tag:blogger.com,1999:blog-8277474234931401119.post-28659138516530969662019-03-25T08:43:00.002+08:002019-03-25T10:08:40.136+08:00Azure Cloud, an annoying mess of random errors & warningsContext: We recently moved to Azure from AWS. We did this for business reasons. It's been about 2 months since we moved.
This would have not been a big deal if Azure wasn't a buggy P.O.S mess.
A few examples:
1. Random errors when completing a merge request. What the fuck is Status Code 0: error error?
Status code 0: error error? #WTF
2. You use Git flow? Well MS says 'fuck you'. You Jaypax Ginetehttp://www.blogger.com/profile/17271980777872608420noreply@blogger.com0