Evolution to Node Scripts

The first question

There must be something that can control email more powerfully, and even create pdfs. What do I need to learn in order to write code that controls computer tasks, not just the browser?

Nodemailer

This led to discovering Nodejs, where I used nodemailer to automate sending emails.

const sendThese = JSON.parse(fs.readFileSync('sendThese.json', 'utf8'));
const listofEmails = JSON.parse(fs.readFileSync('studentEmails.json', 'utf8'));
let incr = 1;
let didntMake = [];
// Nodemailer and gmail would prevent sending bulk emails from default accounts, 
// the workaround I found was to send only 12 at a time.
for (let cert=0;cert<12;cert++){
...    
const emailBody = `<div>Dear ${toName},<div>
 <br>
<div>We are pleased to send the attached Certificate of Completion for the Wilmette Institute course: ${cleanCourse}. ...`
...
  if(result){certEmail = {
    from: 'Quddús George',
    to: result.email,
    subject: `Wilmette Institute Certificate`,
    cc: '*****@wilmetteinstitute.org',
    html: emailBody,
    attachments: [
        {
            filename: `${toName}.pdf`,
            path: `/Users/path/to/certificate/${today}/${cleanCourse}/${toName}.pdf`
        }
    ]
};
} else{
    certEmail = {
        from: 'Quddús George',
        to: 'myself@wilmetteinstitute.org',
        subject: `Can't find email for: ${toName}`,
        cc: '',
        html: `Email Missing for ${toName}`,
        attachments: [
            {
                filename: `${toName}.pdf`,
                path: `/Users/path/to/certificate/${today}/${cleanCourse}/${toName}.pdf`
            }
        ]
    };
    let unfound = sendThese.newCertificates[0]
    didntMake.push(unfound)
    incr = incr+1;
};
sendThese.newCertificates.shift();
transporter.sendMail(certEmail, function(error, info){
    if(error){
        console.log(error);
    } else {
        console.log('Email sent: '+ toName + info.response);
    }
});
}};
sendThese.newCertificates = [...sendThese.newCertificates, ...didntMake];
fs.outputJsonSync('sendThese.json', sendThese, 'utf8');

Asynchronously

I would run this script manually, as to not trigger the bulk email limits. Asynchronous code 😉.

While still rugged, this need and question based learning represented a huge surge forward in my learning. I added certificate generation with pdfkit alongside the emailing. Previously I was studying toward a far off goal, now in rapid bursts of enthusiasm solving one problem or task after another.

Puppeteerjs

Things really changed when I discovered Puppeteerjs, created and opensourced by Google.

Breaking News

Now that the backlog of communication was up to date, I was tasked with manually checking the near 60 moodle course forums to see who had completed courses on a weekly basis. This led me to web scraping.

With Puppeteerjs I was able to start scraping the Wilmette Institute's moodle site for an updated list certificate earners.

async function searchEach(whichPage, ) {
    await page.goto(whichPage, {
        waitUntil: 'networkidle2' // or networkidle0
    });
// Get each course title and a link to its respective page.
    const nameLinkList = await page.$$eval(
        '.coursename',
        (courseLinks => courseLinks.map(link => {
            const a = link.querySelector('.coursename > a');
            return {
                name: a.innerText,
                link: a.href
            };
        }))
    );
// Iterate through the courses
    for (const { name, link } of nameLinkList) {
        await Promise.all([
            page.waitForNavigation(),
            page.goto(link),
            page.waitForNavigation({ waitUntil: 'networkidle2' })
        ]);
// Certificate Earners are identified by those who have completed their "Self Assessment"
        let [button] = await page.$x("//a[contains(., 'Self')]");
        if (button) {
            const forumHref = await page.evaluate(el => {
                return el.href;
            }, button);
            console.log(forumHref)
            SAlinkData.links.push({ course: name, forumLink: forumHref })
            await button.click();
            await page.waitForNavigation({ waitUntil: 'networkidle2' })
        }
        else {
            //If we are not able to find a link to the self assessment forum, make a note to edit the moodle course.
            fs.appendFile('broken links.csv', '"' + name + '"' + ',"' + link + '", Broken\n');
            brokenLinks.push('"' + name + '"' + ',"' + link + '"')
        }
        var elementExists = await page.$$('.discussion');
// If anyone has posted on the forums grab their name and a link to their post.
        if (elementExists) {
            await page.waitFor(500);
            for (let z of elementExists) {
                const studentName = await z.$eval('td.author.align-middle > div > div.align-middle.p-2', a => a.innerText);
                const sALink = await z.$eval('.topic a', a => a.href);
                if (!facultyNames.includes(studentName)) {
                    fs.appendFile('finals.csv', '"' + name + '"' + ',"' + studentName + '",' + sALink + '\n');
                    dataObj.selfAssessments.push({ course: name, student: studentName, link: sALink });
                };
            };
        };
    };
}

Now I could collect the information and maintain updated records without the tedious process of clicking through the courses, waiting for courses to load, and copy-paste.

A series of scripts

Data was stored in JSON and CSV files and scripts were run in order.

Scrape for a list of faculty (To keep my records updated)
Scrape for a list of student emails (^)
Scrape the courses for new certificate earners
Compared to previous data to create a list of new sendouts
Review results manually, remove false positives
Create pdf Certificates
Send out by email (12 at a time)

Still quite primitive, but almost entirely automated, my scripts took about 7 minutes to run. Now more time was freed, I would clock out faster and use the extra time to study. I estimate that about 100 hours of work was saved with this particular collection of scripts.

Nodejs All the Things!

From here on out I did every work task assigned to me with Nodejs scripts. My colleagues were nervous, but the results spoke for themselves, communication was now on time, human errors from hours of copy-pasting were eliminated, and there was time to focus on the next task.

As one task finished I was able to pick up another, and quickly all the paperwork was done in a succinct re-useable manner. From generating fincancial agreements for faculty to duplicating moodle courses for the coming year.

Organization

I also gained a deep appreciation for the importance of orangized and modularized code. Concepts I had come accross, but didn't feel very serious until I had a single folder with some 30 or so JS files and duplicate code all over the place.

Versions and Updating

This was further highlighted as I learned better ways to do things, such as search a page for elements containing a certain text. Now some scripts would be using the refined approach and others the old approach.

The first question

#Nodemailer

Asynchronously

#Puppeteerjs

Breaking News

#A series of scripts

Nodejs All the Things!

#Organization

Versions and Updating

Nodemailer

Puppeteerjs

A series of scripts

Organization