Wednesday, November 11

An IT Director's Perspective on Troubleshooting

Knot So Complicated

I am a big advocate of keeping things as simple as necessary.  The old abbreviated adage, KISS (Keep it Simple, Stupid), still beats like a drum in my head after all these years managing and administering systems.  

After having returned from 10 days of time away from work, I was able to gain perspective on a concept that appears simple, but is one we don't always act on appropriately.  This is the concept of flexible troubleshooting. (And as a brief side note, I cannot advocate enough the importance of taking a break from work every so often.)  During my time away, I was able to completely unplug and let my brain focus on new challenges and puzzles, specifically, fly fishing in the Rockies of Colorado.

While I was off on my fishing expedition, which many would agree is not an overly complicated endeavor in itself, I found that on many occasions I came to a fork in the road that required that I make a decision that significantly affected my time and resources.  This decision (or problem) manifested itself in the form of a knot in my fishing line.  I have no idea how the physics of it play out, but when fly fishing, it is awful darn easy to get your line tangled in a monstrous and unforgiving "bird's nest" of a knot.  The decision I am faced with then is whether to attempt to fastidiously work out the knot, or to cut the line and start fresh with all new rigging.  There are pros and cons to each, and each have a direct relationship on the amount of resources you expend or preserve.

Out of frustration (and compounded by the blazing heat, biting insects, and my general lack of patience), I often cut out the tangled mess and started fresh.  This, however, turned out to be inefficient and wasteful.  I neglected to acknowledge the amount of time I had already spent rigging up the now knotted setup, not to mention the cost of the materials I just threw away.  So I then made the decision to make an attempt to untie every subsequent knot (which there were many).  I quickly realized this was not necessarily the best approach either.  At some point, attempting to work out an extremely tight and complicated knot has diminishing returns.  In other words, the time it would take to figure out how to untie the knot far exceeds the amount of time if you started over from scratch.  And factoring in my time investment (time = money, after all), one can argue there are appropriate times for hunkering down and attempting to work out the problem, as well as times where it makes more sense to start anew.

Over the course of my career in technology, I have faced many similar knots. When a PC gets infected by a virus, does it make more sense researching and attempting to clean the virus, or is it a better use of time and resources to just wipe the PC clean and reinstall everything from scratch?  Or when dealing with a problem with our firewall where network packets were being reconstructed out of order, is the best solution to wipe the entire firewall configuration clean and start over, or do we take the time to engage tech support in an attempt to resolve the issue using the existing configuration?

Most issues in technology are not so cut and dry, and the path to resolution involves more than one fork.  The key here is that when dealing with challenges (whether it be writing code, analyzing network protocols, or untying knots in my fishing line), it is important to remain flexible.  At some point during your analysis of the issue, being flexible affords you the ability to gain the presence of mind to know which fork in the road to take.  If you determine that working out the complication is the best path for your problem, start by attempting to resolve the issue in "bite-sized chunks."  For me this meant finding the end of my fishing line and painstakingly working it in reverse through all the twists and turns until the line was free.  It is a slow and meticulous process that can preserve the resources you have already spent.  And on the flip-side, if you determine it is more appropriate to start from scratch, don't forget to factor in the resource investment that has already been committed.

My goal was to catch (and release) fish, and in the end I was successful because my flexible approach turned out to be not so complicated.

Author: Mike Rasmussen
Editors: Gina Shaw, Kelly Stacy, Heather Tipple Yang

me (1).jpg

Mike Rasmussen is Director of Information Systems. He has been at Edmunds for 9 years and in the IT industry for 20 years. His current projects include overseeing the Edmunds upcoming office move, including the selection of a new corporate phone system and finding a corporate storage solution.

Monday, September 21

Find Your Inspiration and Start Coding

Technologists have the power to impact nearly everyone’s lives. The problem is, it’s not always easy to relate to those unfamiliar with or living outside the realm of technology, but it is important to try.

image (1).png
As a kid, I would spend most of my days playing video games, not really caring what anyone else was doing or what was going on in the world around me. In addition to being a sort of lackluster way to live, this lifestyle also prevented me from getting exposure to an entire world of problems (not necessarily a bad thing), which in turn prevented me from ever getting inspired to solve said problems (<- bad thing). Luckily, my time at college led me into a complete lifestyle overhaul. Now I am both more aware of, and more capable of solving, a multitude of problems, and can shift my focus to getting out there and getting inspiration.

Being in software engineering though, I’ve noticed that many of the engineers I know tend to focus their efforts within their field of study. There is certainly something to be said for becoming a master of a single trade, and I’m always happy to use the latest software contributions, but it’s also too easy to get stuck in a singular way of thinking. Software engineers make up only about 1% of the world’s population, and there are a lot of areas outside of software that could use a little technological innovation. It just takes a little outside inspiration. What activities have you participated in recently that you thought could use improvement? What conversations have you had lately with people who could use a little more technology in their life?

image (2).png
Maybe take a cross-country train ride to find your inspiration. You’ll meet a lot of interesting new people (unless you try really hard to avoid everyone for days), and see a lot of interesting new places. In a plane, you’re separated from the places you’re flying over, as well as from the other passengers (with the exception of the snoring fat guy next to you).

At, developers can get the opportunity to do ride-alongs with salespeople or dealers, giving them exposure to a part of the business that they otherwise may not have known about. It’s good practice in general to know the full pipeline in which you work, as it may prompt you to make smart choices on existing projects due to your new understanding your company’s clients. In addition though (and perhaps more importantly), the experience may get you thinking outside the box to come up with an entirely new product your company hadn’t yet considered.

Maybe you’d rather dig a little deeper into the idea space, and do something most engineers might not do, spend a week with a farmer, in a program such as WWOOF. You may be surprised at the opportunities you see to make an impact, not to mention it would be a welcome break from sitting at a desk watching your muscles atrophy all day.

Perhaps you’re not the entrepreneurial type, you’re just looking for inspiration in your current job. I find inspiration from Rodney Mullen, skater turned Silicon Valley spokesperson. He was discovered by technologists during an interview about skating, and was convinced to make a TEDx presentation linking skating and technology. He presented on the similarities between hacking and skating, and has continued to speak on the subject since, including talks about embracing failure, and the impact of skate inventions on American culture.

The moral of the story is, don’t get so caught up in the day-to-day of your own world. There are lessons to be learned from all walks of life, whether those lessons are technological or personal. Find a company with ROWE (Results Only Work Environment) or equivalent to work for. Take the opportunities it provides to work remotely (ie. while riding a train), take time off to work on a farm, or attend a tech talk from a person in an unrelated field.

Find your inspiration.  Get out there.

image (3).png

Author: Matthew Nease
Editors: Gina Shaw, Kelly Stacy, Heather Tipple Yang

Matthew Nease was hired by straight out of UC Irvine, where he graduated in Computer Science and Engineering. He is currently working on the advertising technology team to automate and optimize our SEM campaign spending. He keeps an active lifestyle, primarily through sports (bubble soccer, volleyball, football, tennis, disc golf) and dancing (swing, country line dancing, salsa).  He also enjoys board games, spending time with family, and just about any other activity you can think of.

Monday, August 24

Being Agile in a ROWEnvironment

It has been almost seven years since I started working at and the last 18 months of it was spent doing Project Management. One of the unique responsibilities as an Edmunds’ project manager was to be a consulting Agile coach. This meant that my team and I coached Agile software development in both Scrum/Kanban format for development teams that were struggling to manage their workload, teams that just need a few pointers, and everything else in between. Since the adoption, we’ve iterated our way to success, and made it ours by becoming truly Agile.

Agile was already deeply integrated into Edmunds project culture when leadership in the company started to really focus on creating the best possible work/life balance experience. This is when we were introduced to a culture shift called ROWE (Results-Only Work Environment). This change came with a set of principles when summed up in one sentence was: “Work whenever you want, wherever you want, as long as the work gets done.” While this may seem like a great idea for everyone, those who are familiar with Agile development that haven’t heard of ROWE might already be asking a few questions. As a seasoned Agile coach, I had to be detailed about this adoption, but the devil in the details was much bigger than I thought.

ROWE and Agile development by definition have a few contradicting principles:

Agile: The most effective form of communication is face-to-face.
ROWE: Work wherever and whenever you want.

Agile: Business people and developers must work together daily throughout the project.
ROWE: Work wherever and whenever you want.

Agile: At regular intervals the team reflects on how to become more effective then tunes and adjusts the behavior accordingly.
ROWE: Work wherever and whenever you want.

While the comparison of principles are meant to be (somewhat) humorous, there is a superficial theme that emerges:

Agile methodology encourages teamwork, while ROWE would seem to emphasize individual results.

What is often missed in the ROWE strategy, is the “R” for “Results”. I have seen time and time again where folks in the team won’t see eye-to-eye on what it means to deliver results. There are principles in ROWE like “all meetings are optional”, that make a lot of folks forget about results. While it may seem like this is not going to work out, the main point - when understood correctly - makes it all come together:

“We win and lose as a team”

The principles in ROWE, while it may seem so, never encourages individualism. ROWE, in textbook terms, is an environment in which the emphasis is on the actual work done. How or when that work is accomplished is less important (source It makes perfect sense in an Agile community as long as we understand the aforementioned point. Yes, this requires some bending of the principles, but Agile - like the name - requires us to adapt quickly to any given situation. While these two working styles may seem counter-productive, both cultures will encourage you to be:

  1. Proactive - take ownership of your own work
  2. An adult - gives you responsibility and holds you accountable
  3. Communicative - requires you to actually talk to your teammates

Below are some of the guidelines for ROWE principles and how we made it work for us

ROWE principles
Edmunds guidelines
  1. People at all levels stop doing any activity that is a waste of their time, the customer’s time, or the company’s time
  1. Think about your actions and assess whether or not this is something that will benefit the company and/or your personal development
  1. Employees have the freedom to work any way they want
  1. Don’t work only when it’s convenient for you. Work together, win together
  1. Work isn’t a place you go, it’s something you do.
  1. “R” in ROWE is for “Results”, not “Remote”. If it’s proven that being in the office to achieve best results, you should be there when necessary
  1. Every meeting is optional
  1. Think about whether or not attending a meeting will help you drive results, and make the right choice
  1. There are no work schedules
  1. Be an adult about when you work. Communicate, collaborate, participate.
  1. No judgement on how you spend your time
  1. Let others know that you will be unreachable, and have a backup plan for time-sensitive issues

We iterated again and again in short intervals until we got it right. Being in a constant “beta” stage made us work out our differences and built even more trust in the team. We saw people happily sacrificing their own time for the greater good of the team. These good deeds were reciprocated with gratitude and trust, not a continued increase in workload. Much of our petty arguments or debates to posture and prove “rightness” fell to the wayside in the collective interest of achieving the expected result because we truly valued this freedom. We achieved success by understanding and fully embracing the true purpose of each culture. This gave us a whole new meaning to agility in being Agile. Most importantly, we understood our own environment and made Agile + ROWE something unique to Edmunds through great cohesion and teamwork.

Author: Shawn Kim
Editors: Gina Shaw, Kelly Stacy, Heather Tipple Yang

Shawn Kim has been driving with Edmunds for more than six years, originally as a data developer, then as a technical project manager. He enjoys various outdoor activities including snowboarding, surfing, biking, and golfing. He works on many different areas of the organization as an Agile coach, innovation manager, and PMO strategist.

Monday, August 17

Is Migration to the Cloud A Financial Win for Your Company?

We will save 70% by migrating to the cloud. There, we said it.  Every post or article on public cloud migration experiences seems to contradict the one I just read.  For every company that claims to have saved money by switching to the cloud, there is another that just moved back to on-premise for the very same reason.  What gives? For background, Edmunds is a privately-held, profitable and growing company.  As 100% of Edmunds’ revenue is derived online, uptime and performance of the website is critical.  We have operated from co-located data centers for 10 years and have lived through multiple build-outs, so we know what it takes to build and run a data center both operationally and financially.  

What steps are involved in preparing to migrate to the cloud? 
A prerequisite to any automation effort is to ensure the underlying processes are sound and sustainable.  Poor choices compound quickly in the cloud.  Many of Edmunds’ applications had to be refactored in order to operate in the cloud.  We took advantage of this work to scrutinize our existing technology stack, down to the OS and database level and questioned whether our choices were extendable to the cloud.  We pursued open source alternatives where practical, and carefully vetted any commercial software to ensure portability and sustainability. What are some things to keep in mind before migrating to the cloud? Find the bodies that are buried in your data center.  When contemplating a migration to the cloud, take the time to accurately assess your current data center costs.  If you are taking an “all-in” approach, as Edmunds did, this exercise is fairly straightforward.  If not, your challenge is to find an accurate method of allocating your data center costs to the applications you will be migrating.  Work closely with your finance team to help develop this cost allocation, but don’t cede the responsibility to them.  Allocations based on revenue or other business metrics will lead you to false conclusions.  It is important that you identify a method to allocate your data center costs by compute resources. I have also seen models where data center costs are derived from typical server configurations, which are projected across business segments.  This doesn’t take into account failed configurations, memory upgrades, and the slick new PDUs that your data center manager had to have. It (almost) goes without saying, but account for all of your data center costs - including monitoring and asset management software, racking and stacking, (re)cabling, even parking costs for your people to spend the day at the data center.   How do I set up my plan for success? Plan for failure.  I’ve heard multiple stories where companies pulled the plug on their cloud strategy due to a surprise monthly bill from their provider.  It is important that you allow your people to experiment in the cloud; set aside a discrete budget and establish accountability measures - these do not need to be hyper-detailed goals, as they will change often.  Identify a single person responsible for tracking and managing your cloud costs (again, don’t outsource this to your finance people). Once you have developed a model to forecast your cloud computing spend that has been presented to your executives and accepted by your CFO, don’t let it collect dust.  Constantly update your forecast as your migration plan evolves and as the market (i.e. pricing) adjusts.  Models that incorporate a blended rate for all of your cloud instances are worthless - take the time to sweat the details. What is the final step in deciding whether or not I should migrate to the cloud? Ask yourself if you’re ready to commit.  Depending on which cloud provider you select, you may have to decide whether committing to a set configuration makes economic (and operational) sense for your organization.  As we learned early on, committing prematurely could cost you more than pay-as-you-go options.  We quickly ruled out longer term commitments as cloud computing, as well as our own business, is evolving far too quickly to place any bets to the contrary. There are other/unpublished ways to negotiate cloud costs lower, however, these generally entail some type of revenue commitment and/or prepayment, which will take you further away from the utility-based model that attracted us in the first place.  As cloud providers are still experimenting with pricing strategies and the market is still searching for its equilibrium, Edmunds values flexibility.  The price wars don’t hurt either. How has Edmunds’ experience been in migrating to the cloud? We are in the home stretch of our migration, and so far, so good.  We run 100% of our production traffic through the cloud, which is now part of our normal environment rotation.  We have retrofitted most of our back-end applications and as we get closer to decommissioning our data center, we will spin-up redundancies in the cloud as well. We’d love to hear from others on how their experiences have been in migrating to the cloud.  

Brad Tawa is Executive Director of Sourcing and Project Management and has been with Edmunds for 7 years.  He has held numerous finance, accounting, sourcing and technology leadership positions throughout his career and played a key role in developing and executing on Edmunds' cloud migration strategy.

Authors: Brad Tawa 
Editors: Gina Shaw, Heather Tipple Yang, Kelly Stacy

Monday, July 27

Self-Governing Code: Using Static Analysis and Automated Testing to Eliminate Word-of-Mouth Standards

by Ben Gibson and Carlos Ruiz

The Problem

Have you ever submitted code for review that is working perfectly and has great unit tests, but the response you get is, "Oh, we use tabs for indentation" or "We use ALL_CAPS_WITH_UNDERSCORES for static variable names. Please change, thx!"?

Even though these fixes are easy to make, it can still frustrating that you didn't know about these standards beforehand.  Or maybe you tried to figure them out by searching the company wiki but you found conflicting results.  Either way, it can be painful that word of mouth was needed to figure out something so trivial.

And It can be even more painful when developers remember different versions of the standards.  When that happens, you get something like this:

Finish changes. Assign for code review.  Implement changes based on first code review.  Submit for second code review.  Second reviewer tells you a different "right" way.  Interrupt everyone's day for a discussion between you, reviewer #1, reviewer #2, the most senior developer on your team.  Agree on something.  Update wikis.  Write an email to the team.  Cross your fingers and hope people read it.  Wash.  Rinse.  Repeat...

Now, that's enough to make you tear your hair out!

The Solution: Never Fear. Automated Tests Are Here!

It doesn't have to be painful.  Many (or all) of the standards that we share by word of mouth or wiki documentation can be enforced with static analysis.

"You want to name your file that way?..." BZZZ! Automated test failure!
"You put a space in between your function name and the open paren?..." BZZZ! Automated test failure!

If You Still Need Convincing

Here are just a few of the benefits of using automated tests to enforce standards.

  • Faster and easier rampup for new developers.
  • Less back and forth during code review.
  • Consistency in code and code reviews.
  • Reviewers can focus on important feedback.
  • Quicker time to market (because of less back and forth).

Examples at Edmunds

These are early examples from our JavaScript where everyone knew the rules/conventions, but they were often overlooked or were not easy to spot.

  • Indenting:  Not 1 tab.  Not 2 spaces.  Not 8 spaces.  4 spaces.
  • Semicolons to end statements: ALWAYS.
  • Global variables:  NOPE.  Not allowed.

Here are more early example where everyone knew the rules/conventions, but because we did not automated tests we always had inconstancies.  The typical excuse for these failures was laziness... I mean time-crunch.

  • Monolithic classes
  • “Paul Bunyan tall” functions
  • Thirty-nine argument functions
  • If and callback nesting and nesting and nesting and nesting... (Pyramid of Doom anyone?)

These are some of the scenarios that we now catch with automated tests.

Going Further

You don't have to stop at trivial tests either.  Why not go farther and implement source lines of code validations (ie. "Your function/class can only X lines long")?  Or you can add cyclomatic complexity checks (ie. "You want me to read _that_ spaghetti code?").  We have recently added these tests to our front end code and it has made our lives and code much more beautiful.

Think of self-governing code as the logical extension of self-documenting code.  Use automated tests whenever possible, and make sure you have tests for any "standards" you put in place.

Ben Gibson has been at Edmunds for one year working as a Front End Engineer. He is currently working on a User Generated Content project that aims to improve user experience submitting reviews for consumers, auto repair services, and dealerships. In his spare time he enjoys recording music and traveling with his wife.

Carlos Ruiz was born and raised in Los Angeles and has rocked the Sr. Front End Engineer position at for the past 5 years. He is currently working on a top secret Edmunds front-end project. Carlos loves all foods, but seafood, and drinks coffee only at Edmunds.

Authors: Ben Gibson, Carlos Ruiz
Editors: Gina Shaw, Heather Tipple Yang, Kelly Stacy

Tuesday, June 23

The Unexpected Bonuses of Microbonuses

During the last year, we started a microbonus program. I was skeptical at first, but surprisingly, receiving microbonuses has increased my happiness at work and helped break down barriers between team-members

What is a microbonus?

A microbonus is similar to a regular bonus you might receive at work, except it is much smaller.

"Why would I want a small bonus?" Well the idea is that you get many small bonuses frequently, and they add up over time.

There are many different ways to implement microbonuses. For us, a microbonus is worth $5. You can give a microbonus to any of your co-workers along with a line of text explaining why you're giving it to them (fun hashtags included).

"Thanks for the help with inject-resource.js and helping me come up with great names for my variables! #teamwork-working-together #teamwork-utilizing-experts #teamwork-supporting-me"

Then, you can redeem them for gift cards at a number of places (Amazon, Best Buy, Target, Donors Choice, etc).

Why are they nice?

As I mentioned before, I was skeptical at first. It seemed like a silly gimmick that no one would use. However, because the increments were so small and anyone could give them out, people started using them frequently.

People started using them to reward others for the little things that never get noticed. For instance, if you take 15 minutes out of your day to help explain something to me, I can say thanks by shooting a microbonus your way.

Other benefits

For me, microbonuses have two surprising benefits:
1. I feel less guilty about "bothering" people.
2. I feel encouraged to go above and beyond for the people I work with.

Now, when I end up taking 20 minutes of someone's time with what I thought would be a 30-second question, I am less likely to feel guilty about it. I know I can thank them for their help in a forum that is visible to everyone and has a monetary reward.

Also, because of the incentive of receiving a microbonus, I find myself wanting to do more things that are beneficial for my team, like documenting issues that have tripped me up or cleaning up code that is outside of the scope of my project.

It's not just about the money

The nice thing about microbonuses is that they're not just about the money (although that doesn't hurt). I get the most pleasure out of being recognized for all the little things that I do for my co-workers on a daily basis. It's nice to know that someone appreciated the 15 minutes I spent helping them, and it makes me feel like they enjoy working with me. And in my opinion, there's nothing like feeling appreciated.

Ben Gibson has been at Edmunds for one year working as a Front End Engineer.  He is currently working on a User Generated Content project that aims to improve user experience submitting reviews for consumers, auto repair services, and dealerships.  In his spare time he enjoys recording music and travelling with his wife.

About the Microbonus program: The Edmunds Technology Behavior Awards and Recognition (aka BAR) pilot program utilizes (a hosted software platform for social recognition) to empower employees to recognize each other for various leadership and teamwork values. The program was conceived to increase employee satisfaction, as well as encourage specific behaviors that boost productivity within the organization.

Author: Ben Gibson
Editors: Kelly Stacy, Heather Tipple Yang, Gina Shaw