Deploying Hundreds of Thousands of SEO Pages with a Few Lines of Code in Rails
When I joined Hired about 3 years ago, we basically had a Home page, About page, and not much else. Recently we’ve decided to dig deeper into SEO and renew our focus on driving organic traffic.
This whole time, I’ve wanted to work on SEO to leverage all of the content we have in a way that would allow us to reach candidates and employers searching for jobs or candidates matching certain criteria. We have tens of thousands of companies on our platform and an order of magnitude more jobs in our database, so we needed to come up with a way to leverage that data.
Looking at the way people search, we decided to start down a path of figuring out how to programmatically build new pages to target searches like “ruby on rails jobs san francisco” or “biotech companies in austin”. We could have gone with a search-like product where filters could be applied as parameters like `/jobs?city_id=1&industry_id=2`, but those urls would be worthless for SEO purposes. We wanted to generate and open up landing pages with urls like `/jobs/san-francisco/biotech`.
The MVPWhen we first began building these pages as an MVP, we started with the most simple version possible: a few lines in the routes file, some constraints, a few new actions, and a bunch of duplicated code. Here’s a basic example:
First we’d add a route:
# config/routes.rb … get '/jobs/:id', to: 'jobs#city_directory', constraints: CityConstraint.new
# lib/constraints/city_constraint.rb class CityConstraint def matches?(request) HIRED_CITY_SLUGS.include?(request.params[:id]) end end
Then in our controller
# app/controllers/jobs_controller.rb def city_directory city = City.find_by(slug: params[:id]) Job.find_by(city: city.name)
end
Pretty straightforward right?
Once we had a basic idea of how these might perform, we decided to go further by adding more top level filters, like state, region, category, industry, required skills, role, and more. So to do that, we started off going the easy route: copy+paste!
All of a sudden our routes file started to look like this:
# config/routes.rb … get '/jobs/:id', to: 'jobs#city_directory', constraints: CityConstraint.new get '/jobs/:id', to: 'jobs#state_directory', constraints: StateConstraint.new get '/jobs/:id', to: 'jobs#industry_directory', constraints: IndustryConstraint.new # [ad infinitum]
This started to look worse when we decided to also filter companies as well, duplicating each line for companies and jobs… The controllers were looking even worse than the routes as we’d create a new action for each filter and write lookup code for each filter combination
def companies_by_role_and_industry… etc
AND we needed to create a new constraint for every single combination… There was duplication everywhere! Yikes!
It Gets WorseThings were already looking messy enough, but then we decided we wanted to build pages with nested filtering like [Find Backend Engineer Jobs at Gaming Companies in San Francisco](https://hired.com/jobs/san-francisco-ca/gaming/backend-engineer)… Did we really need to manually add routes and a controller action for every combination? What about nested filters? Something like
get ‘/jobs/:id/:industry/:role'
started to provide more and more problems. This was clearly no good.
Rethinking What We’re DoingSo I decided to re-evaluate what our SEO pages were actually doing: basically all we wanted to do is take a resource like Job, apply some set of arbitrary filters matching a url pattern like city/industry, and display those with some custom copy here and there for the filter pair, exposing the urls in our sitemap.
So I started by defining a list of url patterns that effectively would also define the filters:
class SeoFinder
def initialize(base_scope, filters={})
@scope = base_scope
@filters = filters
process_filters! if @filters.present?
end
# parse out the filters and apply them with a little meta programming
def process_filters!
@filters.slice(*self.class::ENABLED_FILTERS).each do |filter, value|
send(:"filter_by_#{ filter }!", value)
end
end
def filter_by_city!(city_slug)
city = City.find_by(slug: city_slug)
@scope = @scope.where(city_id: city.id)
@scope.lazy
end
# etc for each type of filter we want, just make sure the queries are uniform across the models you would supply here, which is a great case for adding uniform named scopes!
private
# we can delegate or use method missing to make sure the finder behaves like an AR::Relation
def method_missing(name, *args, &block)
if scope.respond_to?(name)
@scope.send(name, *args, &block)
else
super
end
end
end
Now in our controller method, we just need to forward the request parameters which contain our filters:
# app/controllers/jobs_controller.rb def directory @jobs = SeoFinder.new(Job.active, params) end
Voila!
With this approach, we can now create tons of pages by adding a new pattern to the `/lib/seo_routes.rb` file. If you think about a pattern like `/:market/:industry/:role` and consider we have 16 markets, 43 industries, and 44 roles, just adding this single pattern generates 30,272 new pages!!
Some NotesObviously the code shown above is simplified. In the controller, we also do some stuff like setting instance variable names, and interpolating the variable names into I18n calls to set the meta descriptions and titles and headers, etc. We can also use the `PATTERNS` list to iterate across all the matching records to build our sitemap pretty easily.
