Poltergeist: A PhantomJS driver for Capybara

3 January 2012

This announcement is coming way later than I had originally intended. Last October I started experimenting with the idea of writing a driver for Capybara that would use PhantomJS as the browser.

Initially the biggest problem was addressing the issue of how to communicate between a Ruby process and a PhantomJS process. But then it hit me: PhantomJS gives you a browser environment, so you can do everything you can do in a browser, and you can do Web Sockets in a browser. So I used Web Sockets.

After hacking away for a while I eventually had a pretty complete driver. But I was being plagued by segfaults that were coming from WebKit’s JavaScriptCore JS engine. Thus began months of poking C++ code and getting far too comfortable with gdb.

I tried the Qt 4.8 RC (it has since been properly released) and found that the JSC segfault had gone, but now there was a new segfault. After much hair-pulling I found a workaround. But I was still left with another problem: it wasn’t possible to attach files to <input> elements against Qt 4.8. After yet more hair-pulling I found the culprit for that one, too.

Which leads me to the point where I am now finally happy to invite you, dear reader, to try out my humble little Capybara driver. Let me know how you get on!

Comments

I'd love to hear from you here instead of on corporate social media platforms! You can also contact me privately.

Jon Leighton

2012-01-04 08:57:19 UTC

The build script compiles against the library files in deploy/Qt-4.8/lib, so you could just upload these to your server, plus the phantomjs binary, and make sure that LD_LIBRARY_PATH is set when you launch phantomjs... I think.

Geoff

2012-01-05 07:19:54 UTC

Thanks you. This has come at just the right time for me, I am just about to start the JS part of my project. I'll let you have some feedback once I have used it for a bit.

Jo Liss

2012-01-06 00:33:03 UTC

That looks very cool indeed!

So -- sorry I'm being slow here -- how does this compare to the capybara-webkit gem (performance and functionality-wise)?

Jon Leighton

2012-01-06 10:46:12 UTC

I also made some comments about this in the README of the project but...

* Performance: I'm unable to get the capybara-webkit specs working for me when I tried just now, but last time I tried Poltergeist seemed a lot faster. I am not sure why: there isn't a fundamental reason that Poltergeist should be faster, as they are both built from QtWebKit. Two potential reasons are: 1) I run Linux, there seems to be a performance bug about c-w on Linux, see https://github.com/thoughtb..., 2) Poltergeist's messaging format may be more efficient - poltergeist sends JSON over a web socket, so 1 message per instruction, whereas c-w uses a more low-level format over a (unix) socket that results in several messages per instruction.

* Functionality: should be pretty much identical. They both pass the Capybara suite.

That said, I think there are some additional advantages of Poltergeist:

* It's easier to hack on, as no C++ knowledge is required. The code base is in Ruby and CoffeeScript.

* It can take advantage of advances in the PhantomJS project. For example, as explained in my original post we had to patch WebKit to get it working against Qt 4.8. I think this issue will affect c-w too (but haven't verified). More generally, PhantomJS has a growing community and Poltergeist is able to tap into that rather than having to duplicate the effort in its own C++ code base. Another example is that Poltergeist will produce 'native' click events (rather than triggering DOM events in JS), because that's what PhantomJS does - that means a click won't work if the element is covered up, or is not scrolled in to view, which more closely simulates what a user actually does when the interact with a page. (The scrolling is handled automatically though.)

* Currently PhantomJS is distributing binaries for Windows and OS X. This means the user doesn't have to compile anything themselves, or have Qt on their system. I am working on a binary tarball for Linux too. In version 1.5 PhantomJS intends to have a stripped-down version of Qt in its source tree, which will make it easier to apply patches, and will also mean that PhantomJS can pull in the latest WebKit sources. So basically I think that PhantomJS/Poltergeist is currently easier to install on Windows/Mac than c-w, and I intend that it will become easier to install on Linux soon too.

Despite all the above, c-w is clearly a more mature project at this stage and if it's working for someone on a particular project, I don't think there's a compelling reason to switch. Though I am keen for people to just try it out and let me know if they find bugs :)

Piotr Zolnierek

2012-01-11 22:00:48 UTC

I just tried it on MacOs but it is slower than capybara-webkit (3secs) vs with poltergeist about 5secs. Wonder what could be the problem? I get the impression that the startup time is taking forever... looking at one of the provided examples:

time bin/phantomjs examples/phantomwebintro.coffee
$("#intro").text() -> PhantomJS is a headless WebKit with JavaScript API.
It has fast and native support for various web standards:
DOM handling, CSS selector, JSON, Canvas, and SVG.
PhantomJS is an optimal solution for fast headless testing, site scraping, pages capture, SVG renderer, network monitoring and many other use cases.
PhantomJS is created by
Ariya Hidayat

bin/phantomjs examples/phantomwebintro.coffee 0.78s user 0.17s system 15% cpu 6.014 total

Any idea what that could be?

Jon Leighton

2012-01-11 22:20:30 UTC

Unsure. The binary is packed with http://upx.sourceforge.net/. It might be taking the time to decompress. You could try decompressing it yourself by installing upx and running "upx -d /path/to/phantomjs". I'd be interested in how much difference that makes.

Piotr Zolnierek

2012-01-12 05:35:38 UTC

Can I start and keep phantomjs running in the background??

Jon Leighton

2012-01-12 09:29:41 UTC

You can't attach poltergeist to an already-running phantomjs process, but poltergeist will only start phantomjs once for the whole test run.

Jon Leighton

2012-01-13 16:55:53 UTC

What do you get for just doing `time phantomjs`? Is that also slow?

Could you send me the output of `strace -tt phantomjs`?

Piotr Zolnierek

2012-01-13 17:30:41 UTC

Thanks for looking into that. Sent it by email.

Piotr Zolnierek

2012-01-13 18:13:39 UTC

One more thing.... it seems that visit does not wait until the page is fully loaded as it does with webkit

have a look at this gist
https://gist.github.com/160...

it seems as if phantomjs would not click the button and submit the form, but why?

Jon Leighton

2012-01-13 21:48:38 UTC

That test also fails when I try it with capybara-webkit.

Piotr Zolnierek

2012-01-14 09:53:54 UTC

Can you please add Capybara.current_session.driver.browser.set_cookie and Capybara.current_session.driver.browser.get_cookies??

Piotr Zolnierek

2012-01-14 10:03:35 UTC

Sorry for that... not the best example... The speed issue is not phantomjs related I found out... It seems the speed is exactly the same. The delay was caused because capybara was waiting on the next page element and retrying in the background until it timed out.

I have an internal page with a form, it works with webkit, poltergeist/phantomjs does click the button on my login form, but still the browser cannot login. The session is saved in a cookie. Is it possible that cookies are not retained??

{"name"=>"visit", "args"=>["http://127.0.0.1:3009/stats"]} {"response"=>"success"}
{"name"=>"find", "args"=>[".//*[@id = 'loginBox']", nil]} {"response"=>[0]}
{"name"=>"find", "args"=>[".//*[self::input | self::textarea][not(./@type = 'submit' or ./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox' or ./@type = 'hidden' or ./@type = 'file')][((./@id = 'email' or ./@name = 'email') or ./@id = //label[normalize-space(string(.)) = 'email']/@for)] | .//label[normalize-space(string(.)) = 'email']//.//*[self::input | self::textarea][not(./@type = 'submit' or ./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox' or ./@type = 'hidden' or ./@type = 'file')]", nil]}
{"response"=>[1]}
{"name"=>"visible", "args"=>[1]}
{"response"=>true}
{"name"=>"tag_name", "args"=>[1]}
{"response"=>"INPUT"}
{"name"=>"attribute", "args"=>[1, :type]}
{"response"=>"text"}
{"name"=>"set", "args"=>[1, "anixe"]}
{"response"=>true}
{"name"=>"find", "args"=>[".//*[self::input | self::textarea][not(./@type =
'submit' or ./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox'
or ./@type = 'hidden' or ./@type = 'file')][((./@id = 'password' or ./@name
= 'password') or ./@id = //label[normalize-space(string(.)) =
'password']/@for)] | .//label[normalize-space(string(.)) =
'password']//.//*[self::input | self::textarea][not(./@type = 'submit' or
./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox' or ./@type =
'hidden' or ./@type = 'file')]", nil]}
{"response"=>[2]}
{"name"=>"visible", "args"=>[2]}
{"response"=>true}
{"name"=>"tag_name", "args"=>[2]}
{"response"=>"INPUT"}
{"name"=>"attribute", "args"=>[2, :type]}
{"response"=>"password"}
{"name"=>"set", "args"=>[2, "pwd"]}
{"response"=>true}
{"name"=>"find", "args"=>[".//input[./@type = 'submit' or ./@type = 'image'
or ./@type = 'button'][((./@id = 'sign in' or ./@value = 'sign in') or
./@title = 'sign in')] | .//input[./@type = 'image'][./@alt = 'sign in'] |
.//button[(((./@id = 'sign in' or ./@value = 'sign in') or
normalize-space(string(.)) = 'sign in') or ./@title = 'sign in')] |
.//input[./@type = 'image'][./@alt = 'sign in']", nil]}
{"response"=>[3]}
{"name"=>"visible", "args"=>[3]}
{"response"=>true}
{"name"=>"click", "args"=>[3]}
{"response"=>true}
{"name"=>"render", "args"=>["phantomjs.png", false]}
{"response"=>true}
{"name"=>"current_url", "args"=>[]}
{"response"=>"http://127.0.0.1:3009/sessions/login"}

Jon Leighton

2012-01-14 13:50:23 UTC

See https://github.com/jonleigh.... BTW please feel free to add issues on Github - it's more convenient to track them there than here in the comments.