Sinatra is classy.

My First Sinatra Project

Slowly, but surely, I am working my way through the Flatiron School’s curriculum. Though it is taking me a bit longer than I expected, I also have relocated to Vermont, sold a house, and settled into my new home. So, I guess I shouldn’t be too hard on myself. Living in Vermont is great and I’m hopeful that I can refocus now on getting my Flatiron work done.

To that end, I have completed my first Sinatra application. Sinatara is a framework for web apps that, along with ActiveRecord, can generate some pretty powerful CRUD based apps. Sinatra provides ‘routes’, which are basically pathways to content that your app will follow in response to HTTP requests. These routes allow for loading and manipulating data through Ruby methods.

For my Sinatra app, I imagined a survey system that would allow users to both create and take surveys. I made some decisions along the way to limit the scope of what I would create for this porject. These limitations were selected so that I could fulfill the requirements of the Flatiron curriculum, but also get the project done without going crazy. I’m still going crazy anyway.

To begin, I though about my database and the relationships between the data. I came up with a model that looks something like this:

Sinatra Project Database Model

The “responses” model was not copmleted as previously indicated. So, there are four databases that I used:

So, with that basic model in mind, I got to building the CRUD functionality, models, routes, etc.

To illustrate some of what I learned, let me go over the user authentication and login procedures:

class SessionController < ApplicationController 
  
  get '/login' do
    if !logged_in? 
      erb :'sessions/login'
    else
      @user = User.find_by(email: session[:email])
      redirect :'/'
    end
  end

  get '/registration' do
    if !logged_in?
      erb :'sessions/registration'
    else
      @user = User.find_by(email: session[:email])
      redirect :'/'
    end
  end

  post '/registration' do
     @user = User.new(username: params[:username], firstname: params[:firstname], lastname: params[:lastname], password: params[:password], email: params[:email])
     if !logged_in? && @user.valid?
      @user.save
      session[:email] = params[:email]
      @my_surveys = @user.surveys
      @all_surveys = Survey.all
      redirect '/surveys'
    else
      flash[:message] = ""
      @user.errors.messages.each do |key, message|
        message.each do |text|
        flash[:message] += text + "<br />"
        end
      end
      redirect :'/'
    end
  end

  post '/sessions' do
    login(params[:email], params[:password])
    @user = current_user
    @my_surveys = @user.surveys
    @all_surveys = Survey.all
    redirect '/surveys' 
  end

  get '/logout' do
    session.clear
    redirect :'/'
  end

end

The SessionController manages the user login experience. But three helpers I created in ApplicationController are also relevant:


  helpers do 

    def logged_in?
      !!session[:email]
    end

    def login(email, password)
      user = User.find_by(email: email)
      if user && user.authenticate(password)
       session[:email] = user.email
       session[:user_name] = user.firstname + ' ' + user.lastname
     else
       redirect '/login'
     end
    end

    def current_user
      if logged_in?
        User.find_by(email: session[:email])
      else
        redirect '/' 
      end
    end
  
  end

These helper methods assist the login process by providing the following funtionality:

Now, looking back at SessionController, there are four Sinatra routes that manage the login process:

get ‘/login’ do: This route checks to see if a user is logged in already. If not, it renders the login form. If they are logged in, the user is redirected to their home page.

get ‘/registration’ do: This route renders the registration form if the user is not logged in already.

post ‘/registration’ do: This route handles the registration process. This is where I used validation and flash messages to provide feedback to the user. The validation flash settings are configured in the User class:

class User < ActiveRecord::Base

  validates :username, presence: { message: "You must provide a username." }, uniqueness: { message: "Username is already taken." }
  validates :email, presence: { message: "You must provide an email." }, uniqueness: { message: "That email is alreday taken." } 
  validates :password, presence: { message: "Password cannot be blank." }
  
  has_secure_password validations: false

  has_many :surveys

end

post ‘/sessions’ do: Here, this route logs the user in and then send them to the ‘/surveys’ route where their surveys will be listed for editing, creating, etc.

get ‘/logtout’ do: Finally, this route clears the session hash, logging the user out.

Ok. This post is long and fairly boring, so I’m going to stop here. The CRUD aspect of this app will have to wait for another blog post. That stuff is handled by the SurveyController class. Fun!

Ruby is a precious gem.

Making A CLI in Ruby

For my educational benefit and as part of the process at the Flatiron School, I recently wrote a CLI gem in Ruby. This was a great learning experience and a bit of challenge for a newbie like myself. That said, it was also rewarding and I have gained a much better understanding of things Ruby. Mission accomplished.

My idea was simple, since this was my first effort. So, I decided to make a CLI that searches for a list of doctors in a particular zipcode and then allow the user to get more details about a particular doctor. I cleverly named this app “doctor_finder.” Why? Because it finds doctors!

At first, I thought I would use one of the large, well-known sites, like WebMD or HealthGrades. I also looked at insurance companies websites and government websites. All of them posed challenges that I could not overcome. The commercial sites blocked scrapers, and I wasn’t savvy enough to get around their blocks. Other sites required the user to fill out forms that I could not easily get past in an automated way.

I finally settled on Zocdoc. This website met my needs and allowed me to search for a doctor by zipcode by using a customized URL. It returned 20 doctors to me, that I could parse with Nokogiri.

But first things first. The first step in this whole process was setting up properly for a Ruby Gem. This process is made simple thanks to Bundler. Bundler includes a gem command that creates the directories and basic files you need to get started. It also includes some automation in your gemspec file so you don’t have to manually maintain your list of files. Finally, it sets up a git repo for you. Basically it gets you off to a good start:

bundler gem doctor_finder

Bundler will even ask you if you want to include a license and a code of conduct. The basic structure of a gem includes a bin directory for storing executable files, a lib directory for storing the main code, modules and classes, etc, and a spec or test directory for your testing tools. I didn’t use any tests for this project, but I’m sure I will in the future.

With that done, the next step is to figure out how you will set up your dependencies and files to get things working. Since I was building a CLI, it made sense to start there. I decided that my CLI would call a CLI object and run a method on that object - and that that CLI object would take care of things from then on. So, to get that going, I needed to create a file in the bin directory called ‘doctor_finder.rb’ which would serve as my main executable file. So I also needed to chmod +x doctor_finder.rb to make it executable. Then, I added this code to the new file:

#!/usr/bin/env ruby
#
require_relative "../lib/doctor_finder"

DoctorFinder::CLI.new.call

So, the key line is the DoctorFinder::CLI.new.call. First, I assume a moduel (namespace) called “DoctorFinder” that is actually created by bundler in the versions file. Then I assume an object called CLI, I create a new one, and I call the method call.

So I need to create the CLI class and include a method called call. I’m pasting in the completed object below. This file would be found in the doctor_finder/lib/doctor_finder directory.

# CLI Main file defining the CLI class
#

class DoctorFinder::CLI

  def call 
    puts "\nWelcome to Hooper's Doctor Finder."
    puts "\nWith HDF you can retrieve a list of doctors by zipcode and then get more details about a particular doctor on that list. It's easy!"
    show_list(get_zipcode)
    get_choice_from_list
    farewell
  end

  def get_zipcode
    # Gets a valid zip code from the user
    zip = "" 
    while !iszipcode?(zip)
      puts "\nPlease enter a valid zipcode:"
      zip = gets.chomp[0..4]
    end
    zip
  end

  def iszipcode?(zipcode)
    # Provides a basic level of validation for user input of zipcode.
    if zipcode.length == 5 && zipcode.scan(/\D/).empty?
      true
    else
      false
    end
  end

  def show_list(zipcode)
    # Calls scraper and prints a list of doctors based on the zip code entered by the user.
    DoctorFinder::Doctor.clear
    docs = DoctorFinder::Scraper.scrape_by_zipcode(zipcode)
    docs.each.with_index(1) do |doc, i|
      puts "#{i}. #{doc.name} - #{doc.speciality} - #{doc.city}, #{doc.state} #{doc.zip}"
    end
  end

  def get_choice_from_list
    # Gets a valid choice from the list of Doctors.

    choice = nil 

    while choice != "exit" && choice != "q"
      puts "\n[1..#{DoctorFinder::Doctor.all.length}] Select Doctor | [zip] Start over with new zipcode | [exit] To quit"
      choice = gets.chomp
      if choice.to_i > 0 && choice.to_i < DoctorFinder::Doctor.all.length+1
        doc = DoctorFinder::Scraper.scrape_for_details(DoctorFinder::Doctor.all[choice.to_i-1]) 
        puts "======================================\n"
        puts doc.name
        puts doc.street
        puts doc.city + ', ' + doc.state + ' ' + doc.zip
        puts "--------------------------------------\n"
        puts "Areas of Specialty:"
        puts doc.areas
        puts doc.details
      elsif choice == "zip"
        show_list(get_zipcode)
      end
    end 
  end

  def farewell
    # Tells the user goodbye.
    puts "\n\nThank you for using Hooper's Doctor Finder.  This was an educational experiment, and I learned a lot.  At first it seemed hard, but then it got easier.\n\nSee you next time.\n\n\n\n"
  endj  
end

Let me explain the purpose each method:

call: This is the initial method that is called from the executbable itself. All it does is print some introductory content and then call other methods. First I want to get a list of doctors based on a zipcode the user enters. So, I call a function list_doctors and I pass it get_zipcode. After that, I want to run the main menu loop, and then, when the user is done, say farewell.

get_zipcode: This method simply asks the user for a zipcode and then returns that zipcode.

iszipcode?: This method runs some very basic validation on the user input. The good news is that Zocdoc is very tolerant of user input - so even if they don’t enter a proper zipcode Zocdoc will do its best to return a list of doctors to us.

list_doctors: Prints out a list of doctors based on a zipcode passed to it. In order to accopmlish this, it makes calls to two other classes, Scraper and Doctor. I’ll show you those below.

get_choice_from_list: This is my poorly named main menu loop. Until the user enters valid input, it shows a list of doctors, pulled from the Doctor object. If they choose a valid doctor, it prints that doctors details.

farewell: Says goodbye when the user exits.

Next, lets take a look at the Doctor object. All this class does is store data for a doctor and it maintains an array of all the doctors:

# The Doctor class
#

class DoctorFinder::Doctor

attr_accessor :name, :url, :speciality, :street, :city, :state, :zip, :details, :areas

  @@all = []

  def initialize
    @@all << self
  end

  def self.all
    @@all
  end

  def self.clear
    @@all = []
  end
end

Not much going on here. But if this was part of a larger application, having the Doctor class as a separate class might prove useful. One thing to note - I made a clear method to empty out the array of doctors so I could do a new search in a new zipcode without creating another instance. For the rest of the app, the action is in the Scraper class:

# The Scraper class
#

class DoctorFinder::Scraper

  BASE_URL = "https://www.zocdoc.com/"

  def self.scrape_by_zipcode(zipcode)
    html = Nokogiri::HTML(open("#{BASE_URL}search?address=#{zipcode}&insurance_carrier=-1&day_filter=AnyDay&gender=-1&language=-1&offset=0&insurance_plan=-1&reason_visit=75&after_5pm=false&before_10am=false&sees_children=false&sort_type=Default&dr_specialty=153&"))
    slice = html.css('.js-prof-row-container')
    slice.each do |doctor| # will go through the HTML and create new doctor instances
      doc = DoctorFinder::Doctor.new
      doc.name = doctor.css('.js-profile-link').text.strip.gsub("\n", ' ').squeeze(' ')
      doc.speciality = doctor.css('.ch-prof-row-speciality').text.strip
      doc.url = BASE_URL + doctor.css('.js-profile-link')[0]['href']
      address = doctor.css('.js-search-prof-row-address').text.strip
      doc.street = address.slice(/^\d+[ ][\w+[ ]]+/) # To format the text correctly, had to use some regex
      doc.city = address[/[ ][ ]+[\w+[.]*[ ]]*[,]/].strip.chop
      doc.state = address[/[A-Z][A-Z]/]
      doc.zip = address[/\d{5}/] 
    end
    DoctorFinder::Doctor.all
  end

  def self.scrape_for_details(doctor)
    html = Nokogiri::HTML(open(doctor.url))
    doctor.details = html.css('.profile-professional-statement').text.squeeze(' ')
    if doctor.details.strip == ""
      doctor.details = "No further details were available."
    end
    doctor.areas = html.css('li.specialty').text.squeeze(" ").gsub("\r\n \r\n ", "\r\n").lstrip
    doctor
  end
end

The Scraper class has two methods and one constant. The BASE_URL constant was just a helpful way to desingate the web site I was scraping. The other methods:

scrape_by_zipcode(zipcode) takes a zipcode and scapes Zocdocs for a list of doctors based on that zipcode. I encode the zipcode in a search string nad usethat from my open-uri request. I slice a chunk of html out of the whole site to narrow in on just the doctor info. Then I iterate through the array from Nokogiri tossing the data I want into a new Doctor instance. I use a number of regular expressions and string methods to get only the data I want in the object. I return the array of all the doctors.

scrape_for_details(doctor) takes a doctor and fills in more data about that doctor by scraping the doctor’s detial page. I use the url I scraped before when I constructed the doctor list and pull down the details. Then, like before, I use text methods and regular expressions to get the data I want in the right place. I return the doctor, but now with more details.

So, that’s basically how this things works. As I said at the begninng, I learned a lot playing with this. Now, onward!

Be lazy. Machines should serve human being. Often programmers serve machines unconsciously. Let machines serve you. Do everything you can to make you lazy. (Yukihiro Matsumoto)

Object Orientation and Ruby

Ruby is an object oriented programming language.

When I was in college in the early 90s, OO was a peripheral subject. We learned about object oriented programming, but it was reagarded as a side show. There were special cases and certain applications that benefited from OO, but it wasn’t the main event. We were focused on procedural programming, in highly structured languages like Pascal or lower level languages where we could get real speed advantages like C.

OO has now become a fundamental approach in most high level programming languages. Ruby was developed, in part, to integrate object orientation throuhgout the programming language. Everything in Ruby is an object, for better and worse.

Even a number.

So you can write code like:

1.+(2)

Which takes the object “1”, which is an instance of the Fixnum class, and runs the “+” method on it which adds the argument, in this case, “2”. The result is “3”.

Now, that’s kinda funny, so you can also just write:

1 + 2

and all works just fine. I suppose that is a convenience when dealing with numbers, but it is good to always remember that everything is implemented as an object.

Generally speaking, objects contain both their data structures and their code. In the parlance, the data are called “attributes” or “properties” and the code are called “methods.” On occassion, it is useful to reference the instance of the object in a method, in order to operate on the instance itself. For that, you use the keyword self. Generally, self refers to the current object. But self can also refer to the current class. Ruby uses context to distinguish what exactly self means. So, for example:

class Person

    @name = ''
    @@count = 0
    
    def name
        self.name
    end
    
    def initialize
        self.count
    end
    
    def self.count
        @@count += 1
    end

end

In the example above, self.name refers to the instance variable @name. In that case, self refers to the instance created by initialize. However, in the method declaration def self.count self now refers to the class since using self in a method declaration makes the method a class method.

This is both confusing, and intuitive. It’s confusing because the word ‘self’ is being used to refer to two different things. But it is also intutive, so long as you consider the context. When you are defining a class, at the top level of that class, self should refer to the class itself, like in the method example. But, within an instance method, self naturally refers to the instance of the class, since the method will be used and ‘owned’ by instances.

Well, ok, maybe it isn’t super intutive. And there is a good chance that I got the details a bit wrong here….But that’s the idea.

Reclaiming the Internet (and embracing the well-intended flame)

When I was in collge, back in the early 90s, I was first introdcued to the internet. I had an account on my school’s mainframe, ostensibly for academic reasons, and I had my first ever email address. Back then, no one had email addresses. I could only email my fellow nerd students and a few professors. Soon, I discovered Usenet, and I followed a number of newsgroups religisouly, especially the one’s that commented on beer in a world before micro-brews.

In the early 2000s the internet began to mature. Soon, websites were a thing. As were companies trying to sell you stuff on websites. With the Web 2.0 evolution, websites became dynamic and databse driven and real commerce could be done. I remember learning the basics of Perl and CGI when that was the number one way to create dynamic web content. It wasn’t pretty, but it worked.

My life took a different turn. I went to seminary and became a pastor, while the internet continued to change and evlove, largley driven by economic forces. But I remember a few moments in the early 90s when the internet was about something else. It was about connecting with people, sharing information for the good of all, and, yes occassionally flaming people for not RTFM.

That said, whenever I was duly flamed, I was spurred to learn more. When the internet was not as mean-spirited a place as it is now, a flame was a helpful slap to encourage you to dig deeper, do your homework, be a contributor and not just a consumer. The internet, back then, expected me to have something to offer - and because of that expectation, usually, there was something to learn, something to gain, something to share.

Now, between Facebook and Twitter, the internet has become about shouting at people. Or narcissistically sharing the highlight reel of your life. Or about tempting our consumerist cravings until they explode in credit card debt.

I wonder if it could be different. Hosting this blog, on a server I pay for, with a set of open source tools that empower me to make the blog, all of which is free from anyone’s ownership or commercial ambition, is a reminder of what the internet can allow. The internet can allow us to freely express ourselves, to engage each other with ideas, to share what we are up to. And it can be a profound community that challenges us to be more thoughtful, more self-critical, and more engaged in meaningful, helpful ways.

Recently, I signed up for micro.blog, a new service that hopes to provide a more open, more accesible alternative to Twitter and Facebook. By providing an accessible framework within which people can share short posts of content, micro.blog enables the internet to resemeble its old self. If Manton (micro.blog’s founder) can find a way to keep micro.blog free of the commercial interests that have diminished the effectiveness of tools like Twitter and Facebook, and if the community continues to evolve in an intentional way, with strong and clear community guidelines, the future could be bright for micro.blog and by extension, the internet.

(Usenet began to decline when it’s servers were overwhelmed by people uploading pornographic images and pirated software binaries. I’m not judging, but good community guidelines are essential to ensure that any network or community lives up to it’s best ideals and original vision.)

The end of net neutrailty threatens new ventures like micro.blog. If it turns out that the only real purpose for the internet is to make some people rich, services like micro.blog and the old internet have no hope. Eventually, it might become necesssary for an entirely new network to be built - a public network that benefits all people. For now, let’s hope that enough of us care enough about the internet to realize it’s potential to really, deeply, meaningfully improve people’s lives.

I find the primary reason I get stuck is because of a pesky syntax error.

Getting Stuck...and Unstuck

Getting stuck is part of life. It’s also part of programming.

As a fairly novice coder, I find the primary reason I get stuck is because of a pesky syntax error. Somewhere I forgot to put a parenthesis, or I forget to properly close a code block. It’s easy to miss these things, and the interpreter is unforgiving.

There are a couple of tools and techniques that I find help when I get really stuck.

These tricks help me move beyond a stuck point. But you know what helps when I’m really stuck? I step away from the desk and take a nice long walk, watch some TV, or (don’t tell) play a video game. Taking a break and doing something unrelated really helps. When I return to the desk, I usually find the problem much more quickly than I would have otherwise, thanks to having a refreshed perspective.