Building a Progressive Web App

The final upgrade was to deploy the certificate maker remotely as a full stack progressive web app.

As you likely already know, our web application has two main parts working together: The client side, which the user can see from the browser, and the server side which is able to run Nodejs scripts from a remote computer. The client side creates a user interface and contains some of the associated business logic using a framework created by Facebook called Reactjs.

When the user pushes a button, such as the client side sends a message to the server side through our API. The server side carries out most of the tasks: sending out emails, creating pdf documents, webscraping for data, etc.

Let's examine some of the applications overarching features and technologies.

Reactjs front-end

React uses a virtual DOM overlay instead of making changes to the DOM directly.

The Dom

The Document Object Model

"The Document Object Model (DOM) is a cross-platform and language-independent interface that treats an XML or HTML document as a tree structure wherein each node is an object representing a part of the document. The DOM represents a document with a logical tree. Each branch of the tree ends in a node, and each node contains objects. DOM methods allow programmatic access to the tree; with them one can change the structure, style or content of a document." - Wikipedia

React's virtual DOM usually takes more energy to load up front, but it focuses on efficient edits, changing only the elements necessary when data changes. This offers a much smoother experience when our application makes multiple server API calls, instead of reloading an entirely new page, just the updated information is changed.

The application needed to allow for manual review and the adding and removing of certificates before send out. So we need to be able to add and remove from the data.

Our Server API

What is an API?

Application programming interface

"An application programming interface (API) is a computing interface to a software component or a system, that defines how other components or systems can use it. It defines the kinds of calls or requests that can be made, how to make them, the data formats that should be used, the conventions to follow, etc. It can also provide extension mechanisms so that users can extend existing functionality in various ways and to varying degrees." -Wikipedia

Our API is served with Expressjs making the server functions available to use from the client-side user interface if the user is logged in and authenticated.

// When the user posts a /make request
app.post('/make', async (req, res)=>{
    try {
        const auth = isAuth(req);
        //check if the user is authenticated properly
        if(auth !== null){
            //if so we can make the pdf documents.
            makeCerts();
            console.log('finished making Certificates')
           let data = await JSON.parse(fs.readFileSync('./worker/sendThese.json', 'utf8'));
           console.log(data)
           // respond with the most recent data now that changes have been made.
           res.send(data)
        }
    } catch (error) {
        console.log(error)
    }
})

Select Component

Students can submit their final Self Assessments to faculty for review instead of posting publicly. In this case we need to create a certificate manually at the request of the faculty.

To save time and energy, and to make sure that entries were accurate and consistent Jed Watson's React Select component was used. It was populated with data from the webscrapes and stayed up to date with the latest student list and courses.

Here is an example course selection dropdown.

let courseList = [
        "Building a World Federation: The Key to Resolving Our Global Crises 08/13/2020",
        "Climate Change 04/09/2020",
        "Bringing Consultation to the Workplace 08/20/2020",
        "Economic Justice in a World of Injustices 10/29/2020",
        "Indigenous Perspectives on the Sacred 03/19/2020",
        "Philosophy and the Bahá'í Faith 12/10/2020",
        "Racism in America: The Most Vital and Challenging Issue 10/10/2019",
        "Sustaining 11 Billion People: Challenges for an Ever-Advancing Civilization 04/15/2019",
        "Writing Biographies and Histories: Recording Stories of People and Places 11/12/2019",
    ]
    <Select options={courseList} />

Search For a Course

Inherent Validity

In this way the input field is both dynamic and has inherent validity. Go ahead, try to get the wrong answer! 😁

Using the same method for the student names, we can ensure that the certificates are automatically sent to the correct students.

Authentication with JSON Web Tokens

For authentication we use JSON Web Token. When the user logs in a post request is sent to the server, the server generates an authentication token, using jwt and a secret contained server-side.

Whenever the user requests sensitive information or private pages and features the authentication token, which is stored in state, is sent back to the server. If the token matches the token placed sent by server upon initial login, the data is sent.

API requests require verification

All API calls require this token verification. If the token does not pass verification, the server does not carry out the requested action.

Worker managing scripts with Chron

The data exported from the server actually comes from the Wilmette Institute's moodle site. In order to keep our data updated Puppeteerjs scripts regularly scrape for data. These are scheduled on a timer using Node-Cron.

As you can imagine, node-cron lets us run cron jobs from nodejs. The syntax is very similar to cron:

var CronJob = require('cron').CronJob;
let getStudents = require('./studentEmails.js')
let getFaculty = require('./facultyEmails.js')
new CronJob('35 7 * * */3', async function() {
    await getFaculty();
    await getStudents();
}, null, true, 'America/New_York');

English Translation

"At 07:35 on every 3rd day-of-week.”

If you're using cron or node-cron crontab.guru is a great resource to get started with it's notation.

In this way, before checking for new certificate earners we make sure that our data is up to date and accurate. Cron manages regular checks for new courses, students, faculty, and certificate earners.

The Google Drive API

This time around, we want to upload any certificates created to the Google Drive folder used for records keeping using the Google Drive API.

Our Google Drive filestructure is as follows:

certificates/year/courseName/studentNameFile.pdf

The certificate folder is consistent, so that is hardcoded in, but the year folder and courseName folders vary so we check Google Drive to make sure the proper folders exist, if not, we create them.

File Id

The Google Drive API uses file ids to describe a files location instead of a unix style filepath, so we have to make an API call to find the parent folder's file id.

function findCourseFolder(parentFolder) {
              //drive.files.list displays a number of files matches the parameters we send through.
            drive.files.list({
              auth: jwToken,
              pageSize: 10,
              parents: [parentFolder],
              //the Google Drive API uses the query string below to filter the results.
              q: `name contains '${courseNameForDrive}' and '${parentFolder}' in parents`,
              fields: 'files(id, name)',
            }, (err, data) => {
              if (err) {
                console.log('Error locating course folder: ' + err);
              }
              let files = data.data.files
              if (files.length > 0) {
                  // If we find the correct folder 
                console.log('Adding to file')
                courseFolderId = files[0].id;
                // upload the pdf file to the filepath we have discovered using Google Drive's file id.
                finalFile(files[0].id)
              } else {
                console.log('No files found.');
                return createFolder(parentFolder)
              }
            });
          }

Now we are all set to deploy our web application.

Reverse Proxy with Nginx

"nginx [engine x] is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev." -nginx

To add some security, we use nginx as a reverse proxy. This way our website is available to the public internet, without exposing our API port for all to see.

Conclusion

This certificate maker has been such an excellent learning process, and will always hold a special place in my heart for having pulled me into Nodejs so early on in my development. I also began moving around large amounts of data. The DOM became a familiar friend with hours of webscraping. Once begun, the process of automating my paperwork-life-workflow became the standard, and aside from replying to emails 🤔 my entire job has been automated away (and parts of other people's 😅).

Copy Paste Everything

Once I happened upon a co-worker who was about to copy and paste over 3000 rows of contact information 🤨, paginated 30 per page 🤨. Friends don't let friends copy and paste that many times 🤨.

#Reactjs front-end

The Dom

#The Document Object Model

#Our Server API

What is an API?

#Application programming interface

#Select Component

Inherent Validity

#Authentication with JSON Web Tokens

API requests require verification

#Worker managing scripts with Chron

English Translation

#The Google Drive API

File Id

#Reverse Proxy with Nginx

#Conclusion

Copy Paste Everything

Reactjs front-end

The Document Object Model

Our Server API

Application programming interface

Select Component

Authentication with JSON Web Tokens

Worker managing scripts with Chron

The Google Drive API

Reverse Proxy with Nginx

Conclusion