Tips on Asking Really Good Questions

Over at StickyMinds, they’ve put together a set of tips on how to ask questions to get answers that improve your software testing. Your first question might be, what does that have to do with good QA and good software testing?

A big part of examining an app, a website, or another piece of software is learning to ask the right questions. By asking the right questions, you learn more about what is expected and unexpected behavior. This helps you figure out what the known (and unknown) boundaries of that product are.

Getting into this mindset is important, not just for software testers but for the people working with them as well. Sometimes you need to ask yourself, what isn’t being tested? What blind spots do we have? What biases do our testers have?

It’s worth it to read all six of the tips, but my favorite are tips 4 & 5: Continue reading “Tips on Asking Really Good Questions”

A new name and a new investor

As you may have noticed, we have a new name: testcloud is now test IO. We’re excited about our new name, we think it reflects who we are and what we do better. We are still the same company and we’re still passionate about testing and excited about helping teams to make software better.

We’re also excited to announce that we’ve raised a new round of funding – Series A – from Turn/River Capital. It’s great to have them on board. You might know them as the folks behind BookFresh (now part of Square), Sucuri, and other great web and SaaS companies.

We’ll use this funding will to expand our team and grow in the U.S. There are a lot of companies out there making software, and we know that test IO can help them do it better.

Michael Downing, the CEO of Tout (one of our first U.S. customers and an online video platform for publishers, content creators and advertisers) had something to say for this occasion:

“Bringing software to market before competitors is critical to our bottom line, but finding bugs before our customers do is critical to our reputation. test IO’s crowd of testers continuously uncovers bugs and usability issues that would normally take much longer to identify internally, allowing us to launch applications on time and worry-free.”

It’s our goal to make everyone’s development process that much faster, more efficient, and worry-free. Today’s economy is driven by mobile apps and software, and developers companies from startups all the way to multinationals are looking for faster ways to test software. test IO provides a platform that developers can manage themselves, yet tap into the speed, quality and efficiency of crowdtesting.

That’s idea the at the foundation of our company – that the smartest and most efficient way to do QA is with continuous testing by humans – and it’s on its way to taking over the world. We’re looking forward to making it the normal way that all software is developed.

If you’re excited about this, get in touch! (We’re hiring)

Continuous Testing Manifesto

Build – Test – Learn… and Repeat!

Continuous testing…

  • combines automated testing and testing with people to ensure full test coverage in real life situations for your products
  • makes the most of computers’ precision and humans’ creativity to find bugs before they can cause damage
  • turns testing with people from the major drag just before shipping into an essential part of development, just like automated testing
  • can be used to solicit tester feedback to improve and shape the product from the beginning of and throughout the development process

Continue reading “Continuous Testing Manifesto”

Mobile Fragmentation Sucks: How to Cope

Device fragmentation is a nightmare for developers of all stripes looking to grow their user base. In fact, it’s the kind of problem that only gets worse the more successful your app becomes. The more users you have, the more diverse and divergent their devices they’re use to run your software will become. These devices vary in terms of form factor, underlying hardware, and operating systems. Even different versions of the same OS present some of the greatest development challenges – Android is a prime example.

Continue reading “Mobile Fragmentation Sucks: How to Cope”

Crowdtesting Tip: Be Specific

One awesome part of working with a crowd of professional human testers: you can turn them loose on your app or website, and they’ll kick the tires and take your product for a spin, to make sure it does what they think it should, and doesn’t do anything it shouldn’t. This is a great way to see what they find and how what you’ve been working on holds up when a normal human uses it.

However, if there are certain types of bugs you’re worried about, new features or functionalities you want checked over, or parts of the site you know you need to refine, take the time to be specific with your testers. Pinpoint the areas you want testers to focus on, so you’ll get the information you need.

The point of testing your product is to help you make it better, so take advantage of being able to specify where you know you need work. Don’t be afraid to be direct about what those are. You can run other test cycles where you leave the test brief open.

Want more crowdtesting tips? Check this post out:

Testing Disaster:

The website wasn’t broken because the government’s programmers are incompetent. It was broken because the government failed to test it enough before releasing it.


A basic rule of programming is that all software will have major bugs. This is as true of the best programmer in the world as it is of the worst programmer. However, all programmers tend to this mistake: only testing completed projects.

Continue reading “Testing Disaster:”

Crowdtesting Tip: Know Your Users

It’s easy to get taken in by the power and flexibility of crowdtesting. Real people can be testing your software or website at a moment’s notice. You can get the right mix of devices, and coverage for the most important mobile, browser, or computer OS versions. It’s a heady feeling, Your team has, up until now, probably been neglecting black box testing in favor of automated and in-house testing (Don’t!It’s not a trade-off you need to make). In the interest of making up for lost time, it can’t hurt to get lots of testers for every release, right?


Don’t take this the wrong way: it’s great to test your product across a huge variety of people and scenarios. However, it’s most important to know who will ultimately be using your app or site. If you don’t know this, take the time to figure it out. Do user surveys, call up a few clients, ask your account managers and front-line customer service people – do your homework on who your users and customers are.

Stop Neglecting Black Box Testing

We’ve established that both black box and white box testing are crucial to the development of great software and websites. Choosing between white and black box is a false tradeoff: they are necessary and complementary.

In the everyday reality development of software, testing takes a backseat to other priorities, like speed and features. In the world of software testing, black box testing gets neglected in favor of white box testing. Why?

White box testing is can be easily automated

  • White box texting doesn’t need to involve other people
  • White box testing happens when the developer has time
  • White box testing means developers don’t have to communicate test objectives to a third party

But we also know that unbiased, motivated, and diverse human testers are best, so white box testing alone isn’t enough. How can development teams incorporate black box testing without giving up the advantages of automated, autonomous, and on-demand testing?

Crowdtesting as Black Box Testing

test IO is centered around the idea that crowdtesting is the best way to get unbiased and heterogeneous people to test software, websites and apps. By making crowdtests on-demand and easy to set up, it can be as easy to integrate black box testing cycles into development workflow as white box tests.

Don’t settle for products that haven’t been black box tested in the real world by real people.

  • test IO manages the black box crowdtesting process
  • test IO finds unbiased, motivated, professional crowdtesters
  • Crowdtesters are there when you need them: even overnight
  • test IO helps clarify your testing objectives

Crowdtesting and test IO remove the barriers to efficient, rapid, and flexible black box testing. No more excuses.

White Box vs Black Box: A False Tradeoff

Now that we’ve looked at what white box testing is and what black box testing is, the natural question is which one is better.

A quick recap:

  • White box testing is when a developer writes tests to check his or her own code. It verifies that specific parts of the product work as expected, and that any changes don’t negatively impact those basic functions.
  • Black box testing is when a person who is unfamiliar with the software’s development checks functions and features of a product. It’s more open-ended and even though the tester probably gets briefed on what to test, how they get there is up to them and depends on their mindset and experience

You probably see where we’re going with this: even if you have 100% code-coverage and a massive set of unit tests, these won’t let you know that users and potential customers can’t find the “Buy Now” button, that your CSS layout explodes in a rotated iPad2, or that users expect the download link to download an image and not a word document. Does that mean you should pick black box testing only?

These two approaches are complementary: your software engineers should definitely be practicing good development habits with unit, integration, and regression tests. Black box testing isn’t a substitute for these. But having human testers doesn’t absolve your team from having to write their own tests either.

In the grand scheme of things, black box testing is often neglected. Product managers are under pressure to ship fast and ship often. Programmers make the mistake of thinking: “All my unit tests are green, this software must do what it should.”

In the end, only real, unbiased testers can tell you if your software is good or not. If you want to save time, effort, and money – make sure both black box and white box testing are integral parts of your development process.