Friday, April 12, 2013

SEO AJAX / HTML5 App _escaped_fragment_ best practices

Escaped Fragment URI Analysis

So due to a little bit of miscommunication between developers, and some code being released late to production we regrettably had three diverging URLs in our content body, canonical url's and sitemap.  For a period of two weeks the spiders crawled -- and we had the opportunity to go back and see what happened.

Sample size: 240914 urls crawled over  14 days 2013/03/30 - 2013/04/12 -- here are the respective dates and # of URL's crawled:

2013/03/30 - 16,375
2013/03/31 - 18,452
2013/04/01 - 17,865
2013/04/02 - 19,431
2013/04/03 - 15,443
2013/04/04 - 4,832
2013/04/05 - 21,053
2013/04/06 - 24,316
2013/04/07 - 27,477
2013/04/08 - 23,862
2013/04/09 - 22,050
2013/04/10 - 15,488
2013/04/11 - 6,396
2013/04/12 - 7,874

We will break down the # of URL's spidered, and also discuss the encoding of the _escaped_fragment_ which outlined by Google - but also appears to have been adopted by Bing, Yandex, Yahoo,  Facebook and others. 

Canonical URL Results (82%)

URLS:  5858
Yandex ( & Facebook ( will fetch:

URLS: 192774
Google & Bing does not escape #v=1 so they will request:
(this is because they --incorrectly-- assume we will escape the canonical URL in the URI)

Sitemap URL (0.53%)
Requested URL:

1284 links crawled 

NOTE: we found this exceptionally low, the sitemap had thousands of files and clearly is effectively not used (when compared to content & body) - of all URL's crawled only only <1% were retrieved.  Google (and other search engines) clearly prefer organic content.

Content URL: (17%)
Requested URL:
40998 total links crawled
40913 by Google
85 from everybody Else

In Canonical and Content URL's Google encodes *SOME* special characters before requesting the _escaped_fragment_ -- but it did not work how we expected.  Specifically GoogleBot leaves characters such as ., ? and = alone - and encodes other characters including ampersand (seriously wtf!)!pagetype?key1=value 1&key2=value-2
would be requested as:

Also of interest was the fact that GoogleBot's IP address was _clearly_ and undeniable accessing API functions which return JSON.  (more on this later).

Wednesday, February 13, 2013

The best one page checkout flow

In response to some suggestions in the community about how to improve the legacy vstore 1PC (One page checkout):

First - I totally agree, the experience could be improved in so many ways I can't even count them.

Before we proceed, let's establish _my_ goals for the 1PC for legacy vstore.  The only goal with the 1PC was to 'acceptably replicate' the existing vstore legacy checkout paths and provide clients a pathway to an app that would let us eventually turn off the legacy checkout, well before we completely turned off the vstore platform (since checkout represents about 50% of the complexity and one of the largest maintenance costs).
It was intended as a way to help clients avoid disruptive change and to ultimately reduce our support costs (in hindsight it will almost certainly never achieve that goal).  The 1PC was initially an experiment with the feasibility of an app based checkout (knowing that it was the most complicated portion of what had to be done -- and after it was finished an app based site would seem easy).  

I make no claims that we intend to build the only perfect one page checkout --- in part because I believe we already have.  Our 1PC is perfect, amazing, and with each iteration it only gets better.  I'm not saying that in a "look at how beautiful my baby is" way -- rather, after 13 years in e-Commerce, I am positive there is no perfect flow and no matter how good something is, it can always be incrementally improved.   I'm saying that *because* IT CAN BE CUSTOMIZED EASILY, and those incremental improvements will always vary by client.   There is no other checkout platform that is easier to customize to specific requirements in part because it's so very simple out of the gate.

In ZoovyLIVE today, I outlined how I want to focus on things that fundamentally shift, and make those things very accessible and adaptable, and how the app framework is well suited for that.

But with specific regard to 1PC - I feel we've released a good solid product that is easily, affordably and infinitely customizable (easily testable as well, which is a HUGE part of checkout) that can be done as part of any app build.   So it's a perfect "David" statue which can be dressed by the client in whatever attire they see fit (which may vary by season, holiday, special event, etc.).   The 1PC works within my target compatibility parameters (it  was never 100% compatibility due to jquery/prototype incompatibilities).  Had I insisted on 100% compatibility, we never would have gotten it released.  

Anyway, that alone warrants/justifies calling the 1PC an amazing step forward and makes it better than anything on the market. The goal in my business isn't to support 570 different versions of the 1PC, and then try to achieve compatibility with roughly 3,000 unique website designs built on a 13 year old rendering system.  Instead, our mission is to have 3 or 4 that are minimal core releases which can be easily/quickly/affordably adapted to fit different business needs.  So please don't wait for us to make a perfect version just for you and include it as a standard feature -- that isn't what we are trying to do.

On a personal note -- I will ask the community to get together and outline the perfect one page checkout flow right after everybody in our community can agree on which religion is the one true/correct religion!  :-)

I don't expect everybody to agree. I am not trying to discover the one true religion or one true best checkout flow. You all run different businesses and you all have needs. But if there is overlap, that's GREAT!!!!  I've suggested that some clients who need/want the same or similar things can and SHOULD pool their resources to create those things. We'll hire/train/manage/monitor developer(s) to create them and "split the check" across those clients. In some cases, to make it easy for the vendors who offer tools (ex remarketing to abandoned carts) build that feature and include it for free/low cost with minimal expertise and experience.   In software, this is a new concept - in a pizza restaurant I think it's pretty common.    Arrange an ad hoc decision management structure, appoint one client the lead, or do it as a community decision (podio is great for things like this. It's just another workspace for that portion of the project).  Don't reinvent the wheel - unless you need a special wheel.

But our work here at Zoovy, and our responsibility, is shifting from trying to make one platform that does everything, to making a platform that is easy to make do *exactly* what you want and is infinitely adaptable, with the help of minimally trained developers.   As far as I'm concerned, the template-based site model has too many limitations, as well as the generic checkout model. I feel this is not where we need to focus.

We may improve the templates, but look for more revolutionary than evolutionary changes.  There is no  shortage of great ideas/features I want to get into the backend right now that will benefit everybody.

So, I'm not saying that we won't be releasing another 1PC. But I feel pretty comfortable saying it's unlikely we would consider offering one to those with compatibility to vstore, which is 1 year into it's 3 year end of life platform.  The vstore 1PC is not customizable because the vstore product requires a developer spend 400-600 hours of one-on-one instruction learning the vstore TOXML format, and SPECL language to be able to connect it to the 1pc app framework. At this point the number of developers in the world who understand SPECL fluently is 2.5, and two of them are focused on building apps.   

So we have crossed that threshold where it's cheaper to just start from an app.  The next checkouts will likely tie to some interactive features of apps and won't be compatible with vstore (watch ZoovyLIVE for a preview of what we're working on).

From _my_ view the 1PC checkout process is:
1. Use HTML to ask customer for data.   <-- whatever/anything you want
2. Upload data in order format to server.  <-- upload as much or as little information as you gather in step #2
3. Server generates order # sends to app. --> yay!
4. App receives order # and thanks customer.  <-- whatever/anything you want

Steps 2 and 3 are Zoovy responsibility - for #2 we need to make it as easy/reliable to send as possible, for #3 we need to make sure we always create an order or send back a meaningful error "ex: go away, we don't want orders from your country".

But for steps 1 and 4 -- let's get a podio group setup, and get a developer working on what you want!

Take care,


Friday, February 1, 2013

New options for securing administrative interfaces - the new device based approach

I originally intended for this to be a community post - but it's length, to properly discuss the subject warranted a blog post.

Dan - wrote in our community:
I've noticed that when I go back to my iPad or iPhone browser (to see if there are any new orders in Zoovy for example) after having been logged in to Zoovy from one or two different PC's during the day (and after many hours or sometimes days) since I used the the iPad or iPhone to check Zoovy, I am immediately logged in and the screen refreshes with the latest data. While this is really nice in my case, it is different than the single session security the system used to have an could be a problem if we were to use someone else's iPad, phone or PC. They would have the ability to login (or refresh essentially) under my login. Again, this is by just going to the tab that already has the super long URL in it, not by going to - I guess one thing we could do is make a habit of logging out?
Security models are always evolving. 

First - yes Dan, what you suggest is timeless advice -- you should ALWAYS click the exit button if there is a reasonable expectation that somebody you don't want to have access to your store that could have access to your device.
We call it "exit" because it's an app, but it has the same effect as logout, it severs the connection (assuming your Internet is still working)

But I would like to address/defend the long session timeout issue.  I think it's fine.
In today's threat model - we are trusting that the end user or organizational security administrator can purchase, and reasonably secure a device.  And that any security administrator will be able to remotely brick the device. (Brick is a cute way of saying "disable" or "render unusable").

I became very aware of this capability a few years ago when my daughter Kimberly had her iphone stolen at the mall. Once she realized it had been stolen she used her friends phone to login and disable the phone, then she put a message on the phone to the person who took it saying to call her at her phone number so she could get her device back and informing them if they did not that she would report it to the police. She tracked the device to it's last gps location and then called Dad (me) and asked me to go get it.
Now I wasn't sure what type of hardened criminal I would encounter I decided it would be best to call the local constabulary and asked them to have an officer meet me there. The phone had been turned off for a few hours, but we knew it's last location. Ultimately I ended up meeting the officer at a Starbucks down the street and he, and two other officer went over to and retrieved the phone. The officer was able to identify the device because of the message on the phone.
The home-owner was cooperative and did not have the phone, but knew who had it and went to get it for the officer in exchange for us declining to press charges. The officer was able to identify it was the correct phone because of the message on the screen. The phone was returned to us by the officer without incident.
Even if the phone had been lost, all photos, music, etc. was all backed up to the cloud - no data would have been lost. This was my first-hand experience in the evolution of threat models.

We are tracking device authentication now (new feature of the app framework) and while we don't *currently* make anybody go through a complicated "new device authentication" process - it is absolutely something that we will be adding, as well as the ability for the security administrator to remotely terminate authentication for a device (and be notified at "wire speed" when new devices attempts authentication).

I feel compelled to say that features like the one I'm about to describe require a very fast internal event messaging system to connect messages from one webserver to another. (Yep -- it's that same messaging system that's giving us all the grief in supply chain these past few weeks)

 Here is an example device authentication message exchange - (messages are necessary because different devices may be connected to different servers).
Anyway here goes:

  • A New device requests permission => message to security system 
  • A Notification broadcast via event system to all security adminstrators via all available channels - sms,push to native phone app, or popup on screen
  • One or more security administrators receives "user xyz is requesting permission to login from device abc named 'Bobs Home' the device appears to be an ipad/browser/unknown - shall i approve?" 
  • Security administrator responds via sms response, touch approve, or click approve in browser.
  • If no security administrator responds within timeout (ex: 10 minutes) then default allow/deny protocol is followed. 
  • A return message is sent to authentication system and accepted/rejected. 
  • Then the remote session is allowed to proceed/fails and remote user is notified via another event message.

Now our original event system which could take 2-3 minutes per message (each trip) would make that process take 10-15+ minutes - which would be a horrible user experience, the new messaging system (again -- the same one that is causing all your supply chain orders issues) is the one that allows us to deliver that new behavior. 

I think we can make that whole experience described above take less than a minute then I think it would be a welcome amount of security.  Of course security administrators who don't want that level of security can leave "allow any new devices to authenticate" open and never be notified. 

Now - In today's threat model - for the most part once a device is authenticated we *should* assume we can trust it. People don't share phones, or pads (as a rule) - hence the long session timeouts.  Trade a daily password, with a strong device setup. 

There should be a configuration setting IN AN APP (not server based) to idle itself out based on user preferences if that's something you need.

But ultimately that's not the real threat these days, it's stolen/lost equipment.
When you're purchasing a new device consider the security -- new devices have really diverse and highly configurable security models (pin pad, pattern, facial recognition, voice, screen saver/passwords) and so the trend in software/services is "don't re-invent the wheel".  Facial recognition can be fooled by devices with 2d cameras, but not devices with 3d cameras.  Future devices will have different authentication / locking protocols based on the location of the device.  (Ex: in a mall, lock fast, at home, lock slow)  -- this will just get easier and easier as the devices continue to get smarter and more aware of their surroundings.

For that reason it really doesn't make sense for each app to re-create it's own security.  Ultimately if the device isn't secure - I can't defend against that.   For example even if a hostile attacker can't use the application - BUT they have physical access to the device they can just as easily install a keystroke logger, etc. and get in that way -- which is why the device authentication is way more important. 

If you lend your laptop/phone/tablet etc. to anybody you should expect to do a full system wipe when it returns. You should not be processing orders on public internet terminals or libraries, or computers in kinkos -- EVER, seriously, it's about the same chance something bad happening as having vigorous unprotected sex with a Bangkok prostitute. 

Once a "clean" device is authenticated - it should be safe to trust at least for most commercial needs (maybe not industrial or military). If the device trust is broken - it's simply a failure of the security protocols that were on the device before it was ever authenticated.   Passwords suck, they don't work and I for one look forward to a day when we don't need them anymore. 

Because of our app based framework I'm excited by the new things we can do -- it will allow us to define the security administrator role even further.
The security administrator should be able to see activity by device, be notified of suspicious activity on a device and be able to remote terminate that session.
A great example of that is if an employee who was fired but had been logged in on his phone/tablet at home.
In addition we are excited because our app framework lets us better describe the roles of employees/access so that a compromised device (token) can't be dropped into a hostile application and use the API rip the business a new a**hole.

I hope you enjoyed reading this as much as I enjoyed writing it.

Friday, January 25, 2013

New Site

Zoovy, the leader in success driven eCommerce software for online businesses, today has released a new app based website.

Thursday, January 10, 2013

SPECIAL ZoovyLIVE 1/10/13 Connect

Welcome to the ZoovyLIVE event for January 10, 2013.
In order to access this ZoovyLIVE event, please follow the link below a few minutes before 9:00am PST.

Send your questions directly to the event via email or Tweet with the links below, or you may ask your questions directly inside the YouTube feed comment area.

We hope you will join us for this exciting and informative event!

Wednesday, November 21, 2012

Kinect support for Zoovy?

On Tuesday @ Zoovy Live I hinted that we might incorporate Kinect capabilities into our product.  A lot of people either missed it, or were left scratching their heads wondering why!??.  A Kinect is a few hundred dollars and plugs into any USB port, it comes with a highly advanced developer library that is easy to incorporate into a variety of retail applications -- very easily.

Kinect is basically a 3d camera with an innovative software library that lets developers identify and track objects.  Objects -- well those could be anything from packages and products, to puppies, to employees handling goods, to customers walking through a retail storefront.  It can identify motion of an object such as a hand sign, gesture, or arm motion.

Google is planning on building something like the Kinect into it's upcoming Glass project.  Glass is one of those things which might be a little ahead of it's time, but it will set the stage for the next evolution of user interface after we move past touch.  In fact a lot of people (myself included) see touch interfaces as merely a transitional step to a voice and motion based interface(s).  This technology has many applications beyond the geeky ones you may be thinking -- for example anybody need to lose a few extra lbs?  Here is a summary of a paper published this week that explains how using augmented reality to visually increase portion size makes you feel satiated more quickly (this is because the stomach has enough calories way before your eyes and mouth are done eating)

Don't underestimate it -- glass could easily be a win simply as a diet craze, trust me there are a lot of chubby nerds out there. :-)

Saturday, November 17, 2012

My Movie Idea: Avengers Meets Google

I wrote this a while ago and sent it to a friend of mine who is in the comic industry, she was too busy to do anything with it -- and so I am only I, but I thought it would be fun to share it with the world. 

I haven't decided if Sergey Brin (Google Co-Founder) is more like Bruce Wayne, or Tony Stark -- but he's apparently training to be a super-hero. 


I think it'd be fun to see him in the next avengers movie! when the world is attacked by demigod from a parallel universe; frankly he is the guy I want on our side.  From the article above the following are a few initiatives he's actively involved in:

*Autonomous Self Driving Cars
*Heads up Displays on Glasses
*Having a computer with technical blue prints and maps of every building
*Space Ships that are mining asteroids

Oh, and when he's NOT working on geeky stuff he's a fitness nut who is intent on being super buff.
So he's basically a real live cross between Bruce Wayne and Tony Stark.  If you don't believe me, then let me point out a few more data points that were not in the article:

This guy is a pilot! He currently owns a fighter jet that he parks on a nasa base (moffet field). Did I mention he's also a bit of a recluse as well?  At the Google I/O conference he introduced all his friends who are clearly very well versed in skydiving, rappelling, and stunt BMX riding!

The comic or movie was an idea for one that loosely parallels the lives of the Google founders (but is set a few years in the future after the two CEO's have come to an impasse and split in their quarrel, one who is corrupted by wall street and pulled to "The Dark Side", one who holds true to the "don't be evil" Google motto). 

Since Sergey has nearly unlimited money, and he is only one lab accident, or love-interest-gone-horribly-wrong from being a super villain!  So the story could be told either way.
Frankly I think it'd be much more fun to watch than "The Social Network."