node-red-twitter-follower

View the Project on GitHub

Node-RED Twitter-follower-flow

Node-RED is very good for prototyping, that is the reason why it is often used in hackathons. If you are new to Node-RED and you start to develop a Node-RED flow, you normally have following challenges:

The Node-RED flow of that project has the objective to provide an (little advanced) introduction to the first three topics above. The CRUD (Create, Read, Update, Delete) topic that is not covered here. Visit that CRUD example for more information.

The YouTube video gives a 13 min introduction to the Twitter-follower-flow.

Introduction to the Node-RED flow

1. The UseCase

The UseCase “Extract the twitter follower list” addresses three topics listed above in the Node-RED Twitter-follower-flow.

  1. Define own REST endpoints to encapsulate Twitter developer API calls.
  2. Automate the authentication the two step authentication of the Twitter developer API.
  3. Extract data from the Twitter follower list and build an own list.

The Twitter-follower-flow is an concrete example of following topics …

The gif shows a execution of the Twitter-follower-flow:

3. Setup the Twitter-follower-flow example

That section contains six major steps to setup the Twitter-follower-flow example on IBM Cloud.

  1. Setup Twitter developer application
  2. Create a Node-RED instance on IBM Cloud
  3. Install a additional Node to the Node-RED instance
  4. Import the Node-RED flow
  5. Configure the Node-RED flow
  6. Introduction to the Node-RED flow (13 min video)

In the image you see the full flow.

Step 1: Setup Twitter developer application

You need a Twitter account and a Twitter developer application. With your Twitter account can register for the Twitter developer API and create a Twitter developer application. That application provides the needed credentials to access the Twitter API, which is used in that example.

Here is a blog post with an example how to setup a Twitter application: How to Register a Twitter App in 8 Easy Steps

Here is a preview, how to get the credentials from your Twitter developer application:

b. Twitter API documentation get-followers-list

Step 2: Create a Node-RED instance on IBM Cloud

We use a Node-RED instance on IBM Cloud with an IBM Lite Account.

  1. Create an IBM Cloud Lite Account just by register here.
  2. Follow the steps in my blog post to setup a Node-RED instance on IBM Cloud

Step 3: Install a additional Node to the Node-RED instance

Install the node-red-node-base64 Node to the Node-RED instance. The gif shows the installation.

Step 4: Import the Node-RED flow

Import the flow from the flow.json file in flows folder of that project. The gif below shows how to import the flow in the Node-RED.

Step 5: Configure the Node-RED flow

We need to configure …

a. Set Twitter API key and secret

Insert the values for the user and secret of the Twitter API credentials in the function set user and secret. The image shows an example, where you get the credentials information.

The mapping for the Node-RED flow:

  1. user = Twitter API key
  2. secret = Twitter API secret key
  3. nodereduser = Your own definition to secure the Node-RED REST Endpoints
  4. noderedpassword = Your own definition to secure the Node-RED REST Endpoints

Here is the source code for the set user and secret function.

var user = "USER";
var secret = "SECRET"
var nodereduser = "admin";
var noderedpassword = "notreallysecure"

flow.set("nodereduser", nodereduser);
flow.set("noderedpassword", noderedpassword);

msg.payload = user + ":" + secret;

return msg;

b. Set Twitter username

Define “username display name” you want get the follower list from, by inserting in function set_basic_auth the "YOUR_TWITTER_DISPLAY_NAME".

// Set basic auth
flow.set("auth", msg.payload);
msg.payload = flow.get("auth");

// Init parameters:
// For more information visit get-followers-list:
// https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list
flow.set("twitterCursor", "-1");
flow.set("twitterResultCount", "50");
flow.set("twitterSN", "YOUR_TWITTER_DISPLAY_NAME");

msg.nodereduser = flow.get("nodereduser");
msg.noderedpassword = flow.get("noderedpassword");

return msg;

c. Set Node-RED URL

Configure the HTTP request nodes. Replace the https://node-red-my-hackathon.mybluemix.net URL with your URL in each HTTP request node:

The image shows how you enter the URL in the HTTP requests node getTwitterFollower-Indicrect.

That image shows HTTP requests nodes to be change.

Step 6: Introduction to the Node-RED flow

The YouTube video gives a 13 min introduction to the Twitter-follower-flow.

Introduction to the Node-RED flow

4. Understand the content of the Twitter-follower-flow

  1. Implementation of REST endpoints in the Node-RED
  2. Automation of the flow to get followers from the Twitter API

4.1 Implementation of REST endpoints in the Node-RED

That is the implementation of the REST endpoints in Node-RED Twitter-follower-flow.

Both Node-RED HTTP endpoints encapsulating the invocation of the Twitter API call. Each endpoint validates the custom defined nodeusername and noderedpassword, before they start to send the REST call of the Twitter API and provide the result.

That is a simplified view of the combination of the Node-RED Nodes “HTTP in”, HTTP request and HTTP response to implement the REST endpoints.

4.1.1 Endpoint to request the followers

This endpoint does encapsulate the invocation of the external Twitter API call followers/list. The Node-RED endpoint very basiclly secured with a noderedusername and a noderedpassword. Only when the simple verification was successful, the Twitter API call can be invoked. The Twitter API invocation is done with authorization of the bearer token and gets a follower list for a user display name in a page size of 50 entries. The image shows the endpoint.

Endpoint (GET): /followers That endpoint has following query parameters:


  //https://[YOUR NODE_RED URL]/followers?screen_name=}&count=}&cursor=}&nodereduser=}&noderedpassword=}

This is the Twitter API invocation in the HTTP node Request twitter follower list, when the simple user validation was successful.


  //https://api.twitter.com/1.1/followers/list.json?screen_name=}&cursor=}&count=}
a. Function node: set bearer auth headers and parameters

That function Node verifies the nodereduser/noderedpassword input of the HTTP request and loads the flow variable bearer_token and collects the query parameter screen_name, cursorand count for the Twitter API endpoint invocation.

// validate the request user and password
if ( msg.payload.nodereduser === flow.get("nodereduser") && msg.payload.noderedpassword === flow.get("noderedpassword")){
 
    // Get variables
    bearer_token = flow.get("bearer_token");
     
    msg.headers = {
         'Authorization' : bearer_token
    }
    
    // Get query parameter
    msg.twitterSN =  msg.req.query.screen_name;
    msg.twitterCursor =  msg.req.query.cursor;
    msg.twitterCount =  msg.req.query.count;
    
    msg.valid_request = 1;
  
} else {
    msg.valid_request = 0;
}
  
return msg;

4.1.2 Endpoint to request the bearer token from twitter

This endpoint does encapsulate the invocation of the external Twitter API call getToken. To secure the Node-RED endpoint very basiclly it validates a nodereduser and noderedpassword. The Twitter API endpoint itself is secured by a basic authentication with a Twitter key and secret. The getToken call requests the bearer token from the Twitter API and saves the token in a flow variable. The Twitter API provides the getToken endpoint. The image shows the endpoint.

Let’s get a deeper look in the function implementation.

In the set header with bearer token function, we load the base64 encoded auth string and we configure the HTTP auth header for the Twitter API request.

// validate the request user and password
if ( msg.payload.nodereduser === flow.get("nodereduser") && msg.payload.noderedpassword === flow.get("noderedpassword")){
  
  msg.headers = {
     'Authorization' : 'Basic ' + flow.get("auth"),
     'Content-Type' : 'application/x-www-form-urlencoded;charset=UTF-8.'
  };
  
  msg.payload = 'grant_type=client_credentials';
  msg.valid_request = 1;
  
} else {
  msg.valid_request = 0;
}
  
return msg;

4.2 Automation of the flow to get followers from the Twitter API

In this part of the flow we fully automate the authentication and extraction of the follower data.

We start that flow simply with pressing the Inject Node start get follower list button. That is the simplified sequence we use:

  1. Get the authorization bearer token from Twitter to access the Twitter API
  2. Get all followers from Twitter using the bearer token ( the challenge is: how implement the pageing thought the followers list from Twitter )
  3. Extract the names of the follower from the follower list

Remember: The authorization for the Twitter API needs a two step Authentication and we protect the Node-RED REST endpoints with a simple authenication.

The image displays the relevant part of the flow.

That are the different main chapters of the automation.

  1. Get bearer token from Twitter to access the Twitter API
  2. Load and build the follower list from Twitter
  3. Verify the end of follower list
  4. Build the final followers list

4.2.1 Get bearer token from Twitter to access the Twitter API

The image displays the relevant part of the flow.

Let’s get a deeper look in the function implementations.

a. Function node: set_basic_auth

In that function we set the following flow variables to prepare the auth for the Twitter API token request.

// Set basic auth
flow.set("auth", msg.payload);
msg.payload = flow.get("auth");

// Init parameters:
// For more information visit get-followers-list:
// https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list
flow.set("twitterCursor", "-1");
flow.set("twitterResultCount", "50");
flow.set("twitterSN", "YOUR_USER_DISPAYNAME");

// Set basic auth
flow.set("auth", msg.payload);
msg.payload = flow.get("auth");
msg.noderedusername = flow.get("noderedusername");
msg.noderedpassword = flow.get("noderedpassword");

return msg;
b. Function: set_bearer_token

In that function we do the initailization of the following flow variables:

// Set token before loop
flow.set("bearer_token", "Bearer " + msg.payload.access_token);
var followers = [];
// Init the follower list for the flow
flow.set("savedFollowers", followers)

return msg;
c. Function: set twitter user and page cursor

In that function we prepare the needed values of the parameters for the HTTP request of the endpoint /followers:


//https://YOU_NODE_RED_URL/followers?screen_name=}&count=}&cursor=}&nodereduser=}&noderedpassword=}

The function source code:

// Set the tweeter request query parameter before the loop

// Data is used in the REST request 
// (Issue Twitter API REST Request (GET))
// -----------------------------------------------------------------------
// https://api.twitter.com/1.1/followers/list.json?
//   screen_name=}&
//   cursor=}&
//   count=}&
//   nodereduser=}&
//   noderedpassword=}

// For more information visit get-followers-list:
// https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list
msg.payload = flow.get("bearer_token")

// Prepare the get follower request parameter
msg.twitterCursor = flow.get("twitterCursor");
msg.twitterCount = flow.get("twitterResultCount");
msg.twitterSN = flow.get("twitterSN");

msg.nodereduser = flow.get("nodereduser");
msg.noderedpassword = flow.get("noderedpassword");

return msg;

4.2.2 Load and build the follower list from Twitter

Let’s get a deeper look in the function implementations.

a. Function: build follower list

That function loads the existing follower list from the flow variable savedFollowers and add the new followers from the current response to that list.

if (msg.payload.errors !== undefined ){
    if (msg.payload.errors[0] !== undefined ){
        // do nothing
    }
} else {
    flow.set("twitterCursor", msg.payload.next_cursor);

    // load the exiting list from the flow variable to be saved later in the Cloudant database
    var savedFollowers  = [];
    savedFollowers = flow.get("savedFollowers");
    
    var users = [];
    // load the response data from the twitter request users
    users = msg.payload.users;

    // build the customized follower list 
    var follower = {};
    for (var i=0; i<50; i++){
        
       if( users[i] !== undefined) {
         follower.username = users[i].name;
         follower.screen_name = users[i].screen_name;
         savedFollowers.push(follower);
         follower = {};
       } else {
           i = 50;
       }
    }   
    flow.set("savedFollowers",savedFollowers);
    followers = [];
}

return msg;

4.2.3 Verify the end of follower list

The switch verifies the payload.next_cursor value, if it is null or not. If the payload.next_cursor is not null, that indicates that more followers can be loaded from the next page of the follower list. With that we need to trigger the start of the Twitter API request for the next follower page.

In that example the follower page size is 50 and it invokes the function set twitter user and page cursor to start once again with the load of the next page.

The image displays the relevant part of the flow.

Let’s get a deeper look in the function implementations.

a. Function move on: Start of the next load of a page.
msg.payload.message = "move on - next page";
return msg;
b. Function end: End the paging and start to build the final follower list.
msg.payload.message = "end, no additional followers";
return msg;

4.2.4 Build the final followers list

That function uses the extracted followers in the flow variable savedFollowers and builds a new follower list. It also adds data to new follower list:

That is the JSON format of the new follower list.

{
  "twitterFollowers": {
    "date": "2020.3.20",
    "time": 1584691605226,
    "count": 2,
    "followers": [
      {
        "username": "user1",
        "screen_name": "user1"
      },
      {
        "username": "user2",
        "screen_name": "user2"
      }]
  }
}

This is the code of the function build followers name list.

var savedFollowers = [];
    savedFollowers = flow.get("savedFollowers");
    
var data = {};
var twitterFollowers = {}; 

var date = new Date();

twitterFollowers.date = date.getFullYear() + "." + 
            (date.getMonth() + 1) + "." + 
            date.getDate();
twitterFollowers.time = date.getTime(); 
twitterFollowers.count = savedFollowers.length; 
            
twitterFollowers.followers = savedFollowers;
data.twitterFollowers = twitterFollowers;


var reset = [];
flow.set("followers", reset);
flow.set("savedFollowers", reset);

msg.payload = data;

return msg;