Monday, April 7

4 Reasons to Deploy Website Testing Framework (WTF) Technology

A brief summary of Edmunds' transition to Website Testing Framework (WTF) written by Daniel Kang, Edmunds' Director of Software Architecture.

At the beginning of 2014, the Software Architecture team at deployed a new Website Testing Framework (WTF) which enables us to run hundreds of parallel experiments in a reliable manner. Since launching WTF, our company has reached a new level of efficiency and confidence in designing, coordinating and measuring our controlled experiments. We wanted to share our experience building WTF, and go over the motivation for choosing an in-house solution.

Motivation 1: Centralize the configuration of all AB tests
We used to have two separate AB testing systems that were tailored specifically for JavaScript and for CMS templates. In both systems, the test configuration was essentially placed around the code block that would generate the AB variants. Since these configuration blocks were not easily visible, it became a tedious exercise to track down all the AB tests. We ran a very large weekly AB test coordination meeting with all Product Owners and Business Analysts to describe, slot, and track all running tests.

The solution WTF provided was simple. All configuration code for both client and server side experiments are standardized and published from a single source. We also created a WTF Dashboard which shows experiments running in each environment. It contains a description of the test, the expected start and end dates, and the pages and components that are being tested.

Motivation 2: Eliminate wait time to run an AB test
One of the biggest bottlenecks we encountered in launching an AB test, was waiting for your test to get slotted to go live. The process for launching an AB test involved asking a Business Analyst to forecast the percentage of visitors required to confidently detect a lift in key metrics during a two week period. If three campaigns each required 33% of visitors, no other campaigns could be slotted.

WTF changed this approach by allowing each visitor to be part of multiple experiments. Initially there was a lot of resistance to this approach due to concerns of cross contamination between campaigns, but we reinforced the fact that strong interactions between campaigns are rare. We still provide the ability to run a campaign in isolation, but this is strongly discouraged for most experiments since this reduces the number of visitors in our test pool.

With WTF we want developers and product owners to come up with testable ideas, and immediately launch their experiments when ready. This allows us to run more tests than ever.

Motivation 3: Auto coordinate AB test campaigns
There are certain pages on our site that every team wants to test. WTF detects when experiments are trying to test the same page, and splits traffic evenly between the overlapping experiments. It also notifies the Product Owner of the overlap so that they can decide whether they need additional coordination. WTF also guarantees that no experiments will be starved of traffic since the traffic split between all overlapping campaigns happens at the beginning of the visitor’s session. 

Motivation 4: Provide end to end audit of our AB testing system
WTF has standardized reports for all live experiments. To ensure that all aspects of the testing system are functioning normally, we regularly run an AA test. Using a 95% confidence interval, we expect no difference in key metrics 95% of the time. This sanity check detects potential issues with the framework as we develop new features.

Conclusion: What’s coming

As we increase our reliance on controlled experiments for various layers of our products, we want to accelerate the detection of winning recipes. We will be adding features to increase the sensitivity of our reports for key user segments. We are also formulating our Overall Evaluation Criteria, so that we can introduce Multi-Arm Bandit to dynamically split traffic between the recipe variants.

Thursday, March 20

Edmunds at SXSW 2014

A brief recap written by Liz Harsh, Edmunds' Marketing Events Manager

For 4 days Edmunds had prime space within the Mashery Lounge at the Austin Convention Center, the hub of SXSW 2014.  

Edmunds was also featured on 2 panels.  John O’Dell, Edmunds’ Senior Editor, sat on the Connected Car Panel panel, and Stephen Gandee, Edmunds’ Vice President of Mobile and Emerging Technologies, sat on another panel and talked about about Edmunds’ ROWE culture.

We held a photo challenge with our signature Edmunds Car Head on the streets of Austin. Best photos of the day won some cool swag.

Our Price Promise Prom themed Happy Hour was very successful and noted as the best party to date. 

Powder blue bow ties, flower corsages, and a perfect DJ'd playlist courtesy of Beats topped off a night of fun!

10,300 = the number of people who visited the Mashery Lounge where our booth was located

1,783 = the number of pieces of branded swag given out

1,000 = the number of people at our Promise Prom

Edmunds is looking forward to having an even larger presence at SXSW 2015!

Thursday, March 6

Edmunds Brings the Outside in at Hackomotive 2014

A brief recap written by Matt May (innovation expert & Hackomotive emcee) on Edmunds' recent Hackomotive 2014 event - a three-day hackathon that brought in 12 teams from across the country to Edmunds HQ to compete in a "Shark Tank"-style format to develop and showcase ideas and products aimed at making car shopping easier. The winners took home $35,000 in cash. 

In the world of corporate innovation, hackathons have moved well beyond the technology-only focus that the word “hack” often conjures up. They've become a valid method of bringing a diverse and passionate group of people together over a short time to solve real world problems and produce a basket of strong ideas.

Today's hackathon is no longer reserved for techies, programmers and coders. Rather, it draws designers, storytellers, marketers and entrepreneurs. The underlying premise is that creativity is a contact sport, and having dozens of talented individuals rub shoulders and put their heads together is bound to produce something profound.

Here at, the only requirement to participate is a passion for changing the world, if only in some small way. As URBANTIMES points out, “A hacker is someone with an all-consuming desire to deconstruct and see new potential within something already established, often within the rigid confines of dominant culture.”

From Hack Days to Ignite! to the annual public Hackomotive, hackathons have become a popular means of creating high-altitude solutions to high-visibility challenges.


Hackomotive 2013 was held over two days, and followed a wide open format: participants first huddled around pre-determined problem spaces, detailing as many issues and pain points. From hundreds of issues, facilitators prepared affinity maps to pull patterns from the myriad, and constructed discrete problem statements.

Individuals then signed up for the problem statement they wanted to tackle, and self-organized into hack teams. The innovation effort began after lunch the first day, and it continued overnight and through noon the next day. Teams presented their solutions late the second day. A panel of judges evaluated each idea against a set of well-defined criteria, and winners were declared.

While high energy, high enthusiasm, and high engagement characterized Hackomotive 2013, the Edmunds team decided to take Hackomotive 2014 to a new level.


Hackomotive 2014 took on a distinctly different feel: far more purposeful, far more competitive, and far more productive, owing to several new twists.

First, it was extended to three days, and it was by application only. Second, it encouraged teams with ideas already under development to enter, and it required some form of prototype concept accompanying a team's application. Third, a series of "gates"--structured surprise challenges combined with a series of pitches designed to guide prototype development, replaced the single final team presentation format of the previous year. Fourth, the panel of judges wore two hats: evaluator and mentor.

Finally, but perhaps most importantly, Hackomotive 2014 narrowed the focus from 2013's entire automotive experience to just the car shopping experience, in keeping with the corporate purpose.

Running in tandem to the team competition was a series of innovation workshops in the form of trend sessions and design thinking training for invited guests.

This handful of key improvements produced a much higher level of excitement, passion, and commitment for all involved. Twelve teams entered, twelve teams finished, and each left the event with a more evolved concept than with which they walked in.


Not knowing exactly what to expect, a fair dose of shock and awe landed on the teams when they were presented with the first two surprise challenges: a 60-second, stand-up, no-prop pitch to the judges, followed by a one-minute video pitch later in the afternoon.

Both challenges were intended to accomplish two things. First, help teams clarify their message, leveraging a time constraint. Second, enable judges to decide which two teams they wished to mentor over the three days.

The pitches ran the gamut, from rusty and raw to funny and fun. But in the end, judges were able to reach consensus on their coaching druthers.

The sixth floor party was a wonderful and welcome reprieve for everyone!


The surprise challenge of the day was a live-event version of Kickstarter: a consumer tradeshow allowing each team to showcase their idea before twenty consumers specially recruited and armed with $15,000 each in fictional "EdBucks" to spend as they wished on their favorite ideas.

Each team felt that this was perhaps the most relevant, real-world challenge of all, and spent the better part of the day building their booths. Adding the creative tension was the fact that more than just bragging rights were on the line: the team receiving the most money would be awarded a competitive challenge in the form of professional resources to give them a true "leg up" on the surprise challenge of the final day.

Team Au.To, who fancied themselves "Google for cars," won first place, followed closely by TireKickers, a team of mechanics that would inspect a car prior to purchase, in much the same fashion as home inspectors do.

To end the day on an entertaining note, and to test the mettle and conviction of team Au.To, they received a Let's Make a Deal twist: an immediate cash prize of $1000, or the "leg up" prize inside a sealed envelope.

The decision was instantaneous: Team Au.To went with the sealed envelope; they were in it to win it!

Day 3

The prize for winning the consumer tradeshow was recognized by all when the surprise challenge of Day 3 was announced: a 1-minute video to be shot and shown as part of the team's final presentation.

Team Au.To won the use of the video crew, kit-and-kaboodle: cameras, lighting, direction and production…a true advantage.

It seemed Au.To was a lock to win Hackomotive 2014. Or were they?

During the evening between Day 2 and Day 3, several teams went back to the drawing board, with some teams completely pivoting to take a new direction based on consumer reaction, and others honing and enhancing their offering.

In the judges' chambers a rather lengthy back-and-forth ensued.  In the end, the judges agreed that in light of the mobile trend and the Hackomotive focus on improving "shopping" versus "purchasing,” CarCode offered the most impactful and relevant concept.  CarCode was declared the $20,000 first place winner;  Au.To the $10,000 second place winner; and Showroom the $5,000 third place winner.   

If there is one word that describes the Hackomotive 2014 experience it is this: Surprise!

To learn more about Hackomotive 2014 visit

Thursday, October 10

Edmunds is Helping Make the Car Shopping Experience a Two-Step Process

A neat and informative piece written by Seth Berkowitz, President and COO of Edmunds, on the initiatives we are taking here at Edmunds to make the car buying process simpler & easier for consumers.  After all, cars are just another consumer why can't we get a price upfront and then just buy it for that price?  Edmunds has the answer.

The Future of Making car buying easier services 59% of U.S. new car shoppers.  Maritz just named the 4th most trusted review site on the Internet, in the company of Trip Advisor, Zagat and Open Table.  And yet, we are not satisfied.  In particular, we at are driven to the point of distraction by two facts: 1. Car shopping is still very painful (further evidenced by surveys continuously showing low trust levels for dealers); and 2. While our own customers trust us a lot, they have told us repeatedly that our service isn’t unique.  Plenty of sites provide fairly comprehensive information these days; the alternatives to are good enough.  
These facts drove us to make a radical shift.  We have gone so far as to change our mission.  We are now determined to make car buying easier.  We further believe that the way to make car buying easier is to foster trust between consumers and dealers and that has a key role to play in this regard.  (In fact, we have already begun to play this role and have amassed data that proves how potent a role it can be.  I’ll share this data a little further on).
Putting on our dreaming hats, we’ve asked ourselves why we can’t make the following promises to our customers: That by belonging to — much like an Amazon Prime customer by analogy — you will have a substantially easier time at the dealership; you will spend less time at the dealership and much more time in the comfort of your home; you will get what you want, from the salesperson you want, in a manner you can trust.  On further reflection, we realized that we can’t think of any reason why these promises shouldn’t come to fruition.
After all this talk, we decided it was time to start somewhere.  We surveyed our customers, comprehensively, in a manner that resembles the U.S. census, and learned that 54% of Americans believe that getting an actual price on a car is their single biggest unmet need.  In this day and age, Americans cannot comprehend that cars should be different than any other consumer good.  Why can’t they hop online, see the actual price, and decide to purchase?
We saw an important role for ourselves here and resolved to launch what we call the Edmunds Price Promise.  In conjunction with our dealer partners, we offer actual prices on hundreds of thousands of cars – in industry parlance, on specific pieces of inventory.  Our dealer partners – in making the Edmunds Price Promise – are promising our customers that they will honor that price when presented with a physical or digital certificate on that car, and that promise is also backed by  We track each and every case to ensure those promises are being upheld.
Since we launched the program in February, here are the results:
  • 60% more consumers are submitting their personal information on in exchange for Price Promise certificates
  • These consumers are looking at 150% more inventory pages
  • 1/3 of all Price Promise deals close within two days of the certificate download
  • Price Promise leads close 15% faster than any other type
  • But most importantly, Price Promise dealers have seen their ratings and reviews scores rise by 10%.  The trust levels and satisfaction levels inspired by the partnership with are tangible and measurable.
We are so inspired with the performance of this product and the difference it is making for our customers that we are working on a suite of promises.  There is no reason that leases, financing and the trade-in process cannot be negotiated through promises originating with our dealer partners to consumers, backed by  When all is said and done, we want our customers to believe they will get, and to actually experience, a substantially easier time buying a car, in a manner they can trust.
This mission – and our early success – has further inspired us to refresh our website, to begin to make car buying easier.  Aesthetically, this refresh is the most authentic representation of our dreams and aspirations we have ever released yet.  Purchasable cars with actual prices are never more than 3 clicks away.  The refresh launches October 15th.  Be sure to check it out!    And also be prepared for December 25th, when we will add to the 7 billion earned and paid impressions we already generate a year with the launch of a national TV ad campaign. 

You can read more posts by Seth Berkowitz on his blog at

Wednesday, August 21 Named a Top Place to Work in Los Angeles for Fourth Consecutive Year Named a Top Place to Work in Los Angeles for Fourth Consecutive Year

SANTA MONICA, Calif. — August 21, 2013 — The Los Angeles Business Journal today recognized, the premier resource for car buying and automotive information, among the top 20 Best Places to Work in Los Angeles in the large company category (+250 employees). The honor marks the fourth year in a row that has been recognized as a top workplace in Southern California.

“With more and more companies competing each year for ‘Best Places to Work’ recognition, you can’t perform well without finding innovative ways to attract and retain top talent,” says CEO Avi Steinlauf. “It’s a testament to both our staff and our corporate culture that we continue to stand out as a top workplace in the Los Angeles Area.” officially ranked #16 in the large company category and today’s award presentation specifically cited its FitCa$h fitness expense reimbursement program as a top employee perk. Other perks added by within the last 12 months include the “Living the Edmunds Way” peer recognition awards program and Get Active team fitness competitions. has long prided itself on being a progressive workplace. Just last year, adopted a Results Only Work Environment (ROWE), in which employees are offered the freedom to work at any time from any location in which they can most effectively deliver their expected results. also allows employees the freedom (subject to certain requirements) to take as much paid time off as desired. These practices are quickly becoming the new buzz among America’s technology executives, and President Seth Berkowitz has been nominated to speak about’s ROWE culture at next year’s South by Southwest Interactive Festival (SXSW) in Austin, Tex. More details on this can be found at
Each year the Los Angeles Business Journal, a weekly publication examining commerce within the LA economy, recognizes the best employers in Los Angeles County. The assessment process reviews company benefits and policies as well as results of an employee engagement and satisfaction survey. The annual assessment is managed by the Best Companies Group. has also been recognized by Los Angeles Business Journal as a leading institution among rising tech companies in Santa Monica’s Silicon Beach.” The company has previously earned national recognition from The Huffington PostBusinessWeekThe Wall Street Journal and Architectural Record.
There are 18 open positions at that can be viewed any time at details about’s “secret sauce” for innovative corporate culture can be found at

About, Inc. is a car-shopping Web site committed to helping people find the car that meets their every need. Almost 18 million visitors use our research, shopping and buying tools every month to make an easy and informed decision on their next new or used car. Whether you’re at the dealership or on the go, we’re always by your side with our  iPhone and iPad apps and our Android App. Our comprehensive car reviews, shopping tips, photos, videos and feature stories offer a friendly and authentic approach to the automotive world. We’re based in Santa Monica, Calif., but you can connect with us from anywhere by following @Edmunds on Twitter or by becoming a fan on Facebook.