Showing posts with label nodejs. Show all posts
Showing posts with label nodejs. Show all posts

Monday, February 10, 2020

Fixing that CosmosDB Error=2: The index path corresponding to the specified order-by item is excluded.

This bug needs three things:

  1. You're using Azure CosmosDB (I know, I don't like it too)
  2. Have a Mongoose query with a sort option against a ..
  3. Field that inside a sub document.
The query option in question is `{ sort: req.query.order || '-metaData.inserted_at' }`. The metaData.inserted_at field is just a date field. MetaData is just a plain object that has a couple of date fields tracking updates, deletes and such. So when you submit the query it spits out the Error=2 response.

CAUTION: Azure CosmosDB has a emulator isn't really helpful here. It will probably point you in a different direction. In my case, I was able to replicate the error and found a fix in where I 'unchecked' the Provision Throughput option in creating the database. That didn't solve the problem on the server.

In fixing this, you have two options:
  1. No sorting in your code. 
  2. Create the index
I went with option #2. A bit of a hassle. I tried CosmosDB as if it's a MongoDB equivalent.

db.getCollection('collectionName').getIndexes();

db.getCollection('collectionName').createIndex({'metaData.inserted_at':-1});

// shorter version
db.collectionName.createIndex({'metaData.inserted_at':-1});

It will be pain if you created an index that's wrong because you have to delete and create it again.

References:
  • https://docs.mongodb.com/manual/tutorial/manage-indexes/#modify-an-index
  • https://docs.microsoft.com/en-us/azure/cosmos-db/index-overview

Friday, August 17, 2018

Cleaning up your global NPM packages

Sooner or later when you're working with any web app, you'll have to deal with nodejs because frontend requirements will force you to do so. And sooner or later, you'll have a mess of npm packages in your global space. You'll want to clean it up but when you issue the command npm list you get a whole mess of packages with their dependencies.

However there's a useful trick you can do that will make npm only list the top level packages without their dependencies.


$ npm list -g --depth 0


Here's a sample result:

The trick is the --depth 0 parameter. It suppresses the listing of a package's dependencies.

We can now tame our unruly global npm packages.

Ha!

Monday, May 4, 2015

Bootstrapping the View Parts

Time to add a face to our project. So, we break out our favorite framework for the designed-impaired programmer, Bootstrap!

Download the stuff and put it in the public/ folder. I made a bootstrap folder in my public folder to keep it organize. Don't forgot to also get jquery. Bootstrap doesn't work without jquery. This is the easy part.

The next part is kinda hit or miss: Jade

Nodejs with Express uses a templating engine to make views. There are a couple of options but jade is the default. I say hit or miss is because some take to Jade like ducklings to water. Some of you might not be ducklings. 

Here's a sample:

nav(class='navbar navbar-default navbar-static-top')
    .container
        .navbar-header
            button(type='button', class='navbar-toggle collapsed', data-toggle='collapse',                    data-target='#navbar', aria-expanded='false', aria-controls='navbar')
                span(class='sr-only') Toggle navigation
                span(class='icon-bar')
                span(class='icon-bar')
                span(class='icon-bar')
            a(class='navbar-brand', href='#')= title
        div(id='navbar',class='navbar-collapse collapse')
            ul.nav.navbar-nav
                li.active
                    a(href='#') Home
                li
                    a(href='#') About
                li
                    a(href='#') Contact

Jade templating language is quite terse to write compared to raw HTML. It also has a few fun parts like includes, conditionals and mixins.

In the Views/ folder, I made a shared folder and move the common stuff like the layout and navbars. This way I can compose the view like lego bricks AND that's where the fun starts.

Clone the repo at Github.

Saturday, April 18, 2015

Finish the REST (Part 2) with Fiddler debugging and testing

We are doing three things for this part: (1) finish coding the REST api (2) Debug with Fiddler (3) Write Frisby test against our API.

Finishing the REST api is the easy part. Make a folder with the path api/v1/ inside the routes folder. Inside the api folder is another folder called v1. I do this just out of habit. APIs will change in even in a production environments. When that happens, I just add a v2 folder. It's just a cheap and easy way to future proof REST API endpoints.

After the folders, create a posts.js file inside.

var express = require('express');
var router = express.Router();
var mongoose = require('mongoose');

// Get the model
var post = mongoose.model('Post');

// GET /api/v1/posts - GET ALL
router.get('/', function (req, res) {
    post.find({}, function (err, posts) {
        res.json(posts);
    });
});

// GET /api/v1/posts/:post_id
router.get('/:post_id', function(req, res) {
    post.findById(req.params.post_id, function(err, post) {
        if (err) res.send(err);

        res.json(post);
    });
});

// POST /api/v1/posts
router.post('/', function(req, res) {
    var newPost = new Post();
    newPost.title = req.body.title;
    newPost.body = req.body.body;
    newPost.author = req.body.author;
    newPost.published = req.body.published;

    newPost.save(function(err) {
        if (err) res.send(err);
        res.json({ message: 'Post created!' });
    });
});

// PUT /api/v1/posts
router.put('/', function(req, res) {
    post.findById(req.params.post_id, function(err, post) {
        if (err) res.send(err);

        post.title = req.body.title;
        post.body = req.body.body;
        post.author = req.body.author;
        post.published = req.body.published;
        post.meta.favs = req.body.favs;
        post.meta.dvotes = req.body.dvotes;
        post.meta.uvotes = req.body.uvotes;

        post.save(function(err) {
            if (err) res.send(err);

            res.json({ message: 'Post updated' });
        });
    });
});

// DELETE /api/v1/posts/:post_id
router.delete('/:post_id', function(req, res) { 
    post.findByIdAndRemove(req.params.post_id, function(err, post) {
        if (err) res.send(err);

        res.json({ message: 'Post deleted!' });
    });
});

module.exports = router;

The whole thing should be fairly easy to figure out since this is the actual CRUD stuff.

Run the project and we should see our web page. Now we move to Fiddler to see around our API. Remember, you can just Postman instead of fiddler.

Start here: The Composer Tab in Fiddler
After you've open Fiddler, just go to the composer tab and add the API url you're interested in. Don't forget the HTTP verb. Press the Execute button on the top right to see the results.

You'll be able to go to the other tabs to see a lot of information.

The Inspectors tab contents is what we are interested in. This is where we see what's inside the reply from our API (or lack thereof if that is what we are expecting).

And finally, tests for our REST API endpoints. I wanted to add tests for our API to ensure behavior and for regression. It just makes sure I don't break the API as we continue working on this little project of ours.

We will be using the Frisby framework. Frisby is made to test REST API on node.js projects.

So, we install Frisby. Open a terminal, navigate to the project folder and type in:

npm install -g --save frisby

That should install the latest frisby version and save it a dependency in our package.json file. When frisby installed, create a folder named spec and inside that another folder named api. Inside these folders, a javascript file named posts_spec.js.

var frisby = require('frisby');
var baseURL = 'http://localhost:1337/'; // just replace if on live server

frisby.create('Status 200 for GET /api/v1/posts is returned')
    .get(baseURL + 'api/v1/posts')
    .expectStatus(200)
    .toss();

frisby.create('Status 200 for GET /api/v1/posts/:id')
    .get(baseURL + 'api/v1/posts/5525f1e3f6a7f3a00b234a09')
    .expectStatus(200)
    .expectJSON({ 'author': 'jaypax' })
    .toss();


Here we have two tests. Feel free to add more.

Now we just need to run it for which we need jasmine-node.

npm install -g --save jasmine-node

After NPM is done, you should be able to run the command in your terminal: jasmine-code spec/api/


Friby results after running the test 
By the way, you need the project to be running in order to run the test.

You basically have two terminals open: 1 terminal running the project and other terminal to run the test from.

Clone the code at the github repo.

Wednesday, April 15, 2015

Coding the REST services (Part 1) - The model

Coding up the REST services part is similar to coding up the CRUD parts for you traditional desktop application. The difference is quite subtle because you'll be also dealing with HTTP verbs - POST, GET, PUT, DELETE. We are going to make the data model first so we have something to interact with.

First step is creating a connection. In our case, we are connecting to MongoLab, so in your app.js file you add the following:

// our libs
var mongoose = require('mongoose');
var uriUtil = require('mongodb-uri');

// our connection function
var connectToMongoLab = function () {
    var username = process.env.MongolabUsername;     // external var for our username
    var password = process.env.MongoLabPassword;     // same for the password

    var mongolabUri = "mongodb://" + username + ":" + password + "@ds035448.mongolab.com:35448/dbhaxspace";
    var mongooseUri = uriUtil.formatMongoose(mongolabUri);
    var options = {
        server: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } },
        replset: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } }
    };
    mongoose.connect(mongooseUri, options);  // connect us to the db with these options
};

connectToMongoLab(); 

mongoose.connection.on('error', console.log);   // if error in connecting to db, dump errmsg to console
mongoose.connection.on('disconnected', connectToMongoLab); // if disconnected just reconnect

// load models
fs.readdirSync(__dirname + "/models").forEach(function(file) {
    if (~file.indexOf('.js')) require(__dirname + '/models/' + file);
});

This should be added before the routes portions. Now for the actual model.

We're using Mongoose to model our data. Think Entity Framework if you're a C# guy, Hibernate (or JPA) if Java and SQLAlchemy for the python guys. (No, PHP gets not love. #Dealwithit)

Create a folder called Model and create a new javascript file in that folder. In our project, our model is called post and it's declared in the post.js file.

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var PostSchema = new Schema({
    title: { type: String, default: '' },
    author: { type: String, default: '' },
    body: String,
    published: Boolean,
    date: { type: Date, default: Date.now },
    geoloc: {
        latitude: { type: Number, default: 0 },
        longitude: { type: Number, default: 0 },
        label: { type: String, default: ''} 
    },
    meta: {
        uvotes: { type: Number, default: 0 },
        dvotes: { type: Number, default: 0 },
        favs: { type: Number, default: 0 }
    }
});
// methods
PostSchema.method({
    findAll : function(cb) {
        return this.model('Post').find({}, cb);
    }  
});
// statics
PostSchema.static({

});
// register
mongoose.model('Post', PostSchema, 'posts');

Mongoose schema's have some pretty nifty features like types, methods and statics that can be part of the model. The whole thing then wraps up when you call the mongoose.model() function to register the model.

Once we have our model registered. We can then call our model into any part of the app.

var mongoose = require('mongoose');
// Get the model
var post = mongoose.model('Post');

We will pick this up on part 2 where I actually show you the REST parts.

Thursday, April 9, 2015

Node is up: hello world

This "Hello world" thing has a long history in programming but very few student programmers really understood it purpose or assumed wrongly what it's purpose.

The "hello world" program is the first thing done on learning new programming languages and it's also the first program done when trying out a newly setup environment to make sure everything is working. This is what we are doing now.
  1. Start by creating our "Basic Node.js Express 4 Application" in Visual Studio.


  2. Run it, point your browser to http://localhost:1337/ and see our web app running. If it runs we are golden and continue to the next set of steps. If not, then you better get to fixing.

  3. Now that we have a known working node.js app. We should commit this to a source repo like github or bitbucket. At the start of this tutorial I did mention I'll be using SourceTree to manage this but you can go old school and use a terminal if you feel that SourceTree is creepy.



    So create a new repository and don't commit any files yet.

  4. We have to create a .gitignore file so we don't commit temporary or intermediate files. We will be using this .gitignore file. So just go to the root of the new repo and copy that .gitignore file there. SourceTree should be able to pick it the new file and use it, "ignoring" files listed in the gitignore..

    Don't forget to commit the files and push it into your repo. Read this if you need more details about pushing into a remote repo.
Now for the lazy, you can instead clone the repo here. Or if you're just lost then follow this tutorial on how to use git.

Don't be afraid to play around with the code. If you screw up, just delete the offending project folder and clone the project again. After cloning, you'll be good again.

Tuesday, March 31, 2015

My summer 2015 MEAN project: Setting up

So let’s get started. my set up:
  1. Visual Studio: community edition with Nodejs tools. I can hear the WTFs from here. Why? Because it’s an excellent IDE for node development, we can get it for free and I know a couple of people who go into a blind rage when I something from the evil empire. Yes, let the hate flow through you.

    But seriously, any text editor will do if you don’t to use VS.
  2. Fiddler. We need something to debug and test our REST API. We could just use a web browser like…...IE (hehehehe) but sometimes we need to drill down to the raw parts to debug. Besides, it’s another free software.

    Postman can be an alternative.
  3. Sign up for Azure. This is the iffy part because you need a credit card. Azure does have a generous free trial and a teacher/student program via DreamSpark. But if Azure rubs you the wrong way then you can do a local VM like turnkey. It’s the same thing. But if you really, really must use something else then there's Openshift or DigitalOcean or Heroku or Amazon. All these services have free trial accounts in some form.
  4. Github via SourceTree. Someone once said, “you’re not a developer unless you have github.” But if you go all ghetto on me, there’s Bitbucket or Codeplex.

    As for SourceTree, it’s another free software for code repos. It works with Git and Mercurial repos. You could opt for a the command line, I won’t judge you. I mean, I can’t judge someone on how they enjoy pain.
  5. Nodejs and NPM stuff. Install node is straight-forward affair. The stuff you have to do after sort of isn’t. The tribe of terminal user will rejoice this part. 
    • ExpressJS. We need it because a node app isn't much without express
    • bower. We need bower because I’m lazy and don’t want to the whole dance of google, download, unpack and copy for web stuff. 
    • mongoose. Because MongoDB database.
  6. Mongolab account. Mongolab does MongoDB as a Service and they have a free plan with 500MB on it to try out. If you don’t want to then you can install MongoDB on your PC or use a VM.
And that’s it. We got our dev environment set.

Sunday, March 29, 2015

My summer 2015 MEAN project: Dormmetita

Dormmetita. That's the name of the app we are building.

I asked a question over at the CDO-ITG FB page a couple of weeks ago what would the community like to read or do over the summer. I had a couple of options for Java, Node and Python. The overall winner though was nodejs. So for the summer I'm doing a MEAN tutorial to build an old app that a couple of guys "brainstormed" about a year back named Dormmetita. Right, +Romar Mayer Micabalo+Paul Michael Labis+Raven Duran?

Technically it's "Dorm me, tita." which is an app to find and post dorm rooms, apartments for students localized to my city, Cagayan de Oro which is a university town. We might throw in reviews but let's keep it simple for now. We can make up stuff as we go.

 I've broken down the tutorial in these topics:
  1. Setting up 
  2. Node is up: hello world
  3. Coding the REST services (Part 1) - The Model
  4. Finish the REST (Part 2) with Fiddler debugging and testing
  5. Bootstrapping the view parts 
  6. Add Angular to the page
  7. Deploy to Azure, because free is nice

So bookmark this post to follow.