Django-celery Error in Calling apply_async() - takes 1 positional arguement but xx were given

This error confused me initially and the Celery documentation wasn't directly helpful. 

 Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "/Users/jaypax/.local/share/virtualenvs/server-9an_1rEM/lib/python3.6/site-packages/celery/app/task.py", line 518, in apply_async  
   check_arguments(*(args or ()), **(kwargs or {}))  
 TypeError: run_scraper_one() takes 1 positional argument but 40 were given  

I called my task as run_scraper_one.apply_sync(args=('keywords here'), countdown=5). 

The run_scraper_one() method is decorated with @shared_app. So this should work. Should. But apparently after digging around: here and here, I figured out that it wants a list or tuple. 

So, the correct way to invoke the task is: 

run_scraper_one.apply_async(("keyword here",), countdown=5)

Fixed.

Scrapy a JS heavy website using Selenium

Scrapy doesn't really like JavaScript heavy websites especially the ones that load the rest of the HTML via a secondary requests using JavaScript.

To overcome this you either use Splash or Selenium. Unfortunately, Splash is no longer supported. It still works but moving forward, it's going to be Selenium. 

The good news here is that Scrapy already supports Selenium via a middleware: scrapy-selenium.

Steps to use scrapy-selenium:

1. Download a Selenium Driver. For example for Firefox get gecko. I'm assuming you have the browser also installed. 

2. Add the following settings to the settings.py file:

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'path/to/gecko'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'path/to/firefox binary'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

For example (on Windows):

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'c:\\Tools\\geckodriver.exe'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'c:\\Program Files\\Mozilla Firefox\\firefox.exe'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

3. In the spiders, just replace the Request() calls to SeleniumRequest()

4. Add a wait_until test on the SeleniumRequest() to would look like:

SeleniumRequest(url=url, 
                callback=self.parse, 
                wait_time=5,
                wait_until=EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.search-results div.search-cell')))

In this one, we wait for the max of 5 secs or until the element, selected by class to be found in the HTML source. After that, we can use scrapy selectors to find the things we want.

So that's it.

Fixing that CosmosDB Error=2: The index path corresponding to the specified order-by item is excluded.

This bug needs three things:

1. You're using Azure CosmosDB (I know, I don't like it too)
2. Have a Mongoose query with a sort option against a ..
3. Field that inside a sub document.

The query option in question is `{ sort: req.query.order || '-metaData.inserted_at' }`. The metaData.inserted_at field is just a date field. MetaData is just a plain object that has a couple of date fields tracking updates, deletes and such. So when you submit the query it spits out the Error=2 response.

CAUTION: Azure CosmosDB has a emulator isn't really helpful here. It will probably point you in a different direction. In my case, I was able to replicate the error and found a fix in where I 'unchecked' the Provision Throughput option in creating the database. That didn't solve the problem on the server.

In fixing this, you have two options:

1. No sorting in your code. 
2. Create the index

I went with option #2. A bit of a hassle. I tried CosmosDB as if it's a MongoDB equivalent.

db.getCollection('collectionName').getIndexes();

db.getCollection('collectionName').createIndex({'metaData.inserted_at':-1});

// shorter version
db.collectionName.createIndex({'metaData.inserted_at':-1});

It will be pain if you created an index that's wrong because you have to delete and create it again.

References:

• https://docs.mongodb.com/manual/tutorial/manage-indexes/#modify-an-index
• https://docs.microsoft.com/en-us/azure/cosmos-db/index-overview